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EXECUTIVE  SUMMARY 


The  fifteenth  meeting  of  the  DOD  HFE  TAG,  hosted  by  the  Navy 
Personnel  Research  and  Development  Center,  was  held  in  San  Diego, 
California  from  5-7  November  1985.  Meeting  in  conjunction  with 
the  TAG  were  the  following  subgroups:  Controls  and  Displays; 

Human  Factors  Engineering  Guide  to  Systems  and  Equipment 
Development;  Human  Factors  in  Logistics;  Human  Factors  Test  and 
Evaluation;  Manned  System  Modeling;  Professional  Education, 
Training  and  Career  Development;  Technical  Society/Industry 
Committee;  Tri-Service  Human  Factors  Standardization  Steering 
Committee;  Tri-Service  Workload  Coordinating  Committee;  User- 
Computer  Interaction;  and  Voice-Interactive  Systems. 

During  the  three-day  meeting,  a  tutorial  on  the  selection  of 
human  performance  measurements  was  presented,  implementation 
efforts  for  the  Army's  manpower,  personnel  and  training  program 
were  discussed  and  an  analysis  center  for  crew  system  ergonomics 
information  was  described. 

Other  briefings  related  to  the  impact  of  spatial 
disorientation  on  aircraft  attitude/control;  HUDs;  and  artificial 
intelligence  and  expert  systems.  An  overview  of  the  human  factors 
projects  at  NPRDC  was  presented  and,  on  the  final  day  of  the 
meeting,  attendees  toured  the  NPRDC  facilities. 

Administrative  Summary 

.  The  Minutes  of  the  Fourteenth  meeting  are  to  be 

distributed  as  soon  as  they  are  released  from  the 
issuing  printshop. 

.  The  Air  Force  will  chair  meeting  16  with  Mr.  Cyrus 

Crites  [Edwards  Air  Force  Base]  serving  as  the  current 
DOD  HFE  TAG  Chair. 

.  The  Army  will  chair  meetings  17  and  18.  Dr.  Michael 
Strub  [ARI  Field  Unit/Bliss]  will  serve  as  the  Army 
Chair . 

.  NASA  will  host  the  sixteenth  meeting  to  be  held  at  the 
Kennedy  Space  Center,  Florida  in  early  May  1986. 

Committee  and  SubTAG  Summary 

Reports  were  presented  by  the  chairs  or  representatives  of 
the  following  subgroups:  Controls  and  Displays;  Human  Factors 
Engineering  Guide  to  Systems  and  Equipment  Development;  Human 
Factors  in  Logistics;  Human  Factors  Test  and  Evaluation;  Manned 
System  Modeling;  Professional  Education,  Training,  and  Career 
Development;  Sustained/Continuous  Operations;  Technical  Society/ 
Industry  Committee;  Tri-Service  Human  Factors  Standardization 
Steering  Committee;  Tri-Service  Workload  Coordinating  Committee; 
User-Computer  Interaction;  and  Voice-Interactive  Systems. 
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Announcements 


The  following  subgroups  have  selected  new  chairs  or  new 
chairs  select: 

.  Human  Factors  Test  and  Evaluation  —  Dr.  James  C. 
Geddie,  USA  Human  Engineering  Laboratory  Field 
Unit/Hood ; 

.  Technical  Society/Industry  Committee  —  Dr.  Mark  M. 
Brauer,  AIIE  Representative,  Lockheed-California 
Company;  and 

.  Voice-Interactive  Systems  —  Ms.  Elaine  Schiller,  Naval 
Ocean  Systems  Center. 


2.  INTRODUCTION  AND  WELCOMING  REMARKS 

2.1  Call  to  Order  —  Mr.  Cyrus  D.  Crites,  TEST-ENAH,  Edwards 
Air  Force  Base,  California 

Mr.  Crites,  DOD  HFE  TAG  Chair,  welcomed  attendees  to  the 
fifteenth  meeting  of  the  TAG  and  expressed  his  appreciation  to  the 
Navy  Personnel  Research  and  Development  Center  (NPRDC)  for  hosting 
the  meeting.  Presentors  were  reminded  to  submit  hardcopies  of 
their  presentations  and  visual  materials  to  the  TAG  Coordinator 
for  inclusion  in  the  Minutes  of  the  meeting. 

Mr.  Crites  introduced  Commander  John  E.  Kohler  of  NPRDC. 
Commander  Kohler,  a  graduate  of  the  United  States  Naval  Academy, 
also  holds  a  master's  degree  in  business  administration  and  served 
as  the  Commanding  Officer  of  NPRDC  until  October  1985.  Commander 
Kohler's  personal  awards  include  the  Air  Medal  and  the  Navy 
Achievement  Medal. 

2.2  Navy  Personnel  Research  and  Development  Center  Welcome  — 
Commander  John  E.  Kohler 

On  behalf  of  NPRDC,  Commander  Kohler  welcomed  TAG  attendees 
to  San  Diego  and  encouraged  them  to  utilize  the  opportunity  to 
tour  NPRDC 's  human  factors  facilities.  He  noted  the  vital  role 
the  human  factors  community  plays  in  system  design  and  recommended 
that  TAG  members  increase  their  awareness  of  maintainability 
issues . 


3.  ADMINISTRATIVE  BUSINESS 

3 . 1  Minutes  of  the  Fourteenth  Meeting 

The  draft  Minutes  of  the  Fourteenth  Meeting  have  not  been 
released  from  the  issuing  printshop.  These  Minutes  will  be 
distributed  as  soon  as  released.  Amendments  to  these  Minutes 
should  be  sent  to  the  TAG  Chair,  Mr.  Cyrus  Crites,  by  15  January 
1986,  for  inclusion  in  the  draft  Minutes  of  the  Fifteenth  Meeting. 
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[As  of  15  January,  the  Minutes  of  the  Fourteenth  meeting  had  not 
been  released  from  the  issuing  printshop. ] 

3 . 2  Service  Caucus  Reports 

Army  Service  Representative,  Mr.  Clarence  A.  Fry  (USA  HEL), 
reported  that  the  group  discussed  the  Army's  MANPRINT  program  and 
whether  or  not  the  manpower,  training  and  personnel  communities 
have  appropriate/suf f icient  tools  to  address  the  issues  raised  by 
MANPRINT;  and  concluded  that,  at  present,  the  answer  is  negative. 
The  group  also  reviewed  the  OUSDR&E  letter  regarding  cockpit 
automation  technology;  and  in  other  business,  selected  a 
representative  to  interface  with  the  Crew  Station  Ergonomics 
Information  Analysis  Center  project.  Dr.  Michael  H.  Strub  is  the 
TAG  Chair  Select  and  the  group  will  select  a  new  TAG  Service 
Representative  at  its  May  1986  meeting. 

Navy  Service  Representative,  LCDR  Thomas  M.  Mitchell  (PMTC), 
announced  that  the  group  had  formalized  the  selection  of  Dr.  John 
O' Hare  (ONR)  to  fill  the  Navy  seat  vacated  by  Mr.  Paul  M.  Linton 
and  LCDR  Mitchell  to  fill  the  Service  Representative  position 
vacated  by  CDR  Larry  M.  Dean.  LCDR  Mitchell  noted  that  the  Navy 
had  petitioned  the  Operating  Board  to  have  the  Service 
Representative  position  extend  until  May  1988,  so  that  the  Navy 
would  not  have  to  hold  another  election  at  the  May  1986  meeting. 
Dr.  O'Hare's  term  will  expire  when  the  Navy  elects  a  TAG  Chair 
Select . 

Air  Force  Representative,  Dr.  Richard  J.  Schiffler 
(ASD/ENECH),  reported  that  the  group  had  discussed  a  number  of 
Service  issues.  In  other  business,  the  group  recommended  better 
Air  Force  coverage  of  the  various  subgroup  meetings:  Dr. 

Schiffler  will  implement  this  recommendation  prior  to  the  May  1986 
TAG  meeting.  A  new  Service  Representative  will  be  selected  at  the 
May  1986  meeting. 

3 . 3  Operating  Board  Report 

Mr.  Crites  noted  that  the  TAG  had  received  a  copy  of  a 
memorandum  sent  by  Donald  I.  Carter,  Acting  Deputy  Under  Secretary 
of  Defense  (Research  and  Advanced  Technology),  to  the  Assistant 
Secretaries  of  the  Army,  Navy,  and  Air  Force.  The  memorandum 
cites  the  productivity  of  the  TAG  as  a  vehicle  for  the  exchange  of 
technical  data  and  coordination  of  HFE  research  among  DOD 
laboratories,  NASA  and  the  FAA.  [See  Attachment  E.I.] 

Mr.  Crites  reported  that  the  TAG  had  received  a  letter  from 
Captain  Paul  R.  Chatelier  (Office  of  the  Under  Secretary  of 
Defense  R&AT)  requesting  assistance  in  obtaining  background 
information  on  human  factors  research  impacting  Cockpit  Automation 
Technology.  The  Operating  Board  discussed  methods  of  obtaining 
the  desired  information  and  determined  that  the  most  efficacious 
approach  would  be  to  utilize  1498s  (work  unit  R&D  effort 
summaries),  cutting  across  service  projects  by  the  use  of 
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keywords.  Mr.  James  Hartzell  (NASA-Ames)  will  provide  the  1498 
data  to  Mr.  Crites  who  will  distribute  pertinent  information  to 
selected  subgroup  chairs  for  their  inputs.  A  summarization  of  the 
findings  is  to  be  sent  to  Captain  Chatelier  in  early  January  1986. 
[See  Attachment  E.2.] 

A  letter  of  appreciation  from  Dr.  Joseph  A.  Birt  has  been 
received  by  the  Operating  Board.  For  a  copy  of  this  letter,  see 
Attachment  E.3. 

NASA-Kennedy  Space  Center  has  offered  to  host  the  sixteenth 
meeting  of  the  TAG.  The  meeting  will  be  scheduled  for  early  May 
1986. 


4.  COMMITTEE  AND  SUBTAG  REPORTS 
4 . 1  Committee  Reports 


4.1.1  Human  Factors  Engineering  Guide  to  Systems  and 
Equipment  Development  (Dr.  K.  Boff)  --  Mr.  Steve  A.  Heckart, 
Armstrong  Aerospace  Medical  Research  Laboratory,  Wright-Patterson 
Air  Force  Base,  Ohio 


View  Graphs  —  Attachment  0 

Mr.  Heckart  noted  that  this  effort  had  been  ongoing  for 
a  number  of  years;  and  that  the  former  Chair,  Dr.  Joseph  A.  Birt, 
had  requested  and  received  approval  from  the  TAG  Operating  Board 
to  solicit  recommendations  regarding  a  potential  HEGED  update  from 
the  National  Research  Council  (NRC).  NRC  responded  to  the 
Committee's  request  and  recommended  that  technology  considerations 
suggested  that  an  electronic  format  would  be  more  suitable  than  a 
printed  handbook  format.  NRC  also  recommended  that  the  Committee 
consider  the  appointment  of  an  editorial  board  to  oversee  the 
effort.  NRC  informed  the  Committee  that  although  it  was  unable  to 
undertake  or  supervise  a  HEGED  revision,  it  would  be  prepared  to 
make  inputs.  According  to  Mr.  Heckart,  Dr.  Kenneth  Boff  will 
provide  a  formal  response  to  NRC. 

In  light  of  the  recommendations  made  by  the  NRC,  the 
group  discussed  its  status  and  concluded  that  serious 
consideration  should  be  given  to  disbanding  the  Committee  since 
its  task  had  been  completed,  i.e.  that  of  investigating  the 
feasibility  of  an  update  of  the  hardcopy  Human  Engineering  Guide 
to  Equipment  Design.  The  Committee  agreed,  however,  that  the  need 
still  exists  for  some  organized  method  of  determining  the  location 
of  human  engineering  data  and  intends  to  discuss  focusing  on 
accessible  databases  and  retrieval  systems. 

£ 

Mr.  Heckart  presented  a  brief  overview  of  Dr.  Boff's 
Integrated  Perceptual  Information  for  Designers  (IPID)  effort. 
Products  expected  from  this  effort  include  a  handbook,  a  data 
compendium,  and  an  automated  database  management  system.  The 
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Handbook  of  Perception  and  Human  Performance  is  scheduled  for 
publication  by  John  Wiley  &  Sons,  New  York,  in  March-April  1986. 
This  two-volume  work  has  45  chapters  and  over  3000  pages.  The 
engineering  data  compendium  prototype  has  been  developed  and 
distributed.  The  final  version  is  expected  to  contain  over  1200 
entries  in  three  volumes  and  should  be  completed  in  the  spring  of 
1986.  The  Designer's  Associate  characterizes  optimal  DOD  design 
processes,  defines  the  tools  and  databases  needed  and  configures  a 
workstation  concept.  The  Designer's  Associate  serves  as  an 
automated  data  management  for  designers  by  encoding,  in  this  case, 
IPID  human  performance  data  and  biotechnical  and  systems  design 
data.  It  also  functions  as  a  "problem-solving"  partner  by  helping 
to  formulate  relevant  questions,  by  tailoring  questions  to  the 
available  information  resources  and  by  devising  retrieval 
strategies . 

In  closing,  Mr.  Heckart  announced  that  a  short  course  on 
human  perception  and  performance  is  scheduled  for  7-11  April  1986 
in  Dayton,  Ohio.  For  additional  information,  please  contact  Dr. 
Boff  at  (513)  255-7596/7602;  AV  785-7596/7602. 

4.1.2  Professional  Education,  Training,  and  Career 
Development  --Mr.  D.  Todd  Jones,  US  Coast  Guard,  Washington,  DC 

According  to  Mr.  Jones,  funding  is  the  major  impediment 
to  the  subgroup's  continuation  of  its  efforts  to  study  what 
constitutes  a  human  factors  engineering  professional.  To  date, 
three  workshops  utilizing  (Dr.  Sidney)  Fine's  Functional  Job 
Analysis  have  been  held: 

Workshop  I  -  HFE  Specialists  in  RDT&E  in  Military 

Systems, 

Workshop  II  -  HFE  Specialists  in  Consumer/Commercial 

Systems,  and 

Workshop  III  -  HFE  Specialists  in  Forensics/Safety. 

Mr.  Jones  will  continue  to  explore  funding  alternatives 
and  will  report  on  his  efforts  in  this  regard  at  the  next  subgroup 
meeting. 


4.1.3  Technical  Society/Industry  Committee  --  Dr. 
Frederick  A.  Muckier ,  American  Psychological  Association/Division 
21  Representative,  Essex  Corporation,  San  Diego,  California 

Dr.  Muckier  noted  that  the  Committee  had  held  two 
meetings  with  eight  organizations  being  represented.  As  is  the 
TS/I  custom,  each  representative  supports  those  SubTAGs  in  which 
his  skills  and  interests  can  be  utilized.  In  attendance  were: 

.  Dr.  Mark  M.  Brauer,  American  Institute  of 
Industrial  Engineers  --  User-Computer  Interaction 
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.  Dr.  Thomas  M.  Cook,  Association  of  Artificial 
Intelligence  (P)  —  User-Computer  Interaction 

.  Dr.  Richard  F.  Gabriel,  System  Safety  Society  — 
Controls  and  Displays 

.  Mr.  Wolf  J.  Hebenstreit,  Electronic  Industries 
Association  and  the  National  Security  Industrial  Association  — 
Tri-Service  Human  Factors  Standardization  Steering  Committee 

.  Dr.  Norman  E.  Lane,  Human  Factors  Society  —  Manned 
System  Modeling 


.  Mr.  Maurice  A.  Larue,  Jr. ,  Aerospace  Industries 
Association  —  Tri-Service  Human  Factors  Standardization  Steering 
Committee 


.  Dr.  Frederick  A.  Muckier,  American  Psychological 
Association/Division  21  —  Human  Factors  Test  and  Evaluation. 

Dr.  Muckier  reported  that  the  Committee  is  attempting  to 
respond  to  the  request  for  data  on  cockpit  automation  technology 
and  he  reminded  TAG  attendees  that  the  Committee  is  available  to 
make  inputs  or  to  provide  data  in  a  variety  of  areas,  should  the 
need  be  called  to  its  attention. 

In  other  business,  Dr.  Mark  M.  Brauer  has  been  selected 
as  the  new  TS/I  Chair. 

4 . 2  SubTAG  Reports 

4.2.1  Controls  and  Displays  --  Mr.  Jeffrey  D.  Grossman, 
Navy  Personnel  Research  and  Development  Center,  San  Diego, 
California 


See  Attachment  F.l  for  the  Minutes  of  the  Meeting.  See 
Attachments  F.2  and  F.3  for  related  materials. 

4.2.2  Human  Factors  in  Logistics  --  Mr.  Dale  Mahar, 
Pacific  Missile  Test  Center,  Point  Mugu,  California 

See  Attachment  G  for  the  Minutes,  Attendee  List  and 
briefing  materials  submitted  by  Mr.  Mahar. 

4.2.3  Human  Factors  Test  and  Evaluation  (CDR  W. 

Moroney )  —  Mr.  Keith  S.  Karn,  Naval  Air  Test  Center,  Patuxent 
River,  Maryland 

See  Attachment  H  for  the  Minutes,  Attendee  List  and 
related  materials  submitted  by  Mr.  Karn. 

4.2.4  Manned  System  Modeling  —  Mr.  E.  James  Hartzell, 
NASA-Ames  Research  Center,  Moffett  Field,  California 
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Charter  and  View  Graphs  —  Attachment  I 

Mr.  Hartzell  reported  that  the  SubTAG  had  discussed  a 
variety  of  models  and  modeling  issues.  However,  due  to  time 
constraints,  it  was  not  possible  to  discuss  the  various  projects 
in  great  depth.  The  subgroup  hopes  to  remedy  the  situation  by 
participating  in  a  proposed  modeling  workshop  to  be  held  in  the 
summer  of  1986  at  Wright-Patterson  Air  Force  Base.  [For  an  update 
on  workshop  plans,  please  contact  Mr.  Hartzell  or  Mr.  Robert 
Bachert  at  (513)  255-2423/AV  785-2423.] 

Brief  overviews  of  ongoing  projects  were  presented  by: 

.  Mr.  Joe  M.  Jones  (AD/ENP;  Eglin  Air  Force  Base) 
discussed  his  efforts  on  modeling  of  voice  sounds.  Although  this 
effort  is  not  a  typical  voice  stress  analysis  model,  it  provides 
some  interesting  information  with  regards  to  workload  and  stress. 

Dr.  Charles  C.  Jorgensen  (Oak  Ridge  National 
Laboratory)  discussed  artificial  intelligence  and  robotic  projects 
at  Oak  Ridge  and  noted  that  Oak  Ridge  is  also  involved  in  the 
modeling  of  the  outer  layers  of  retinal  function. 

.  Dr.  Norman  Warner  (Naval  Air  Development  Center) 
gave  a  review  of  HOS  and  its  applications  to  the  chemical  defense 
area. 


.  Mr.  Michael  D.  McNeese  (Armstrong  Aeromedical 
Research  Laboratory)  presented  a  thorough  report  on  ongoing 
modeling  efforts  at  AAMRL  and  gave  a  general  overview  of  the  Rapid 
Intelligent  Prototyping  Laboratory  (RIPL)  and  the  C  ’  Operator 
Performance  Engineering  (COPE)  program. 

.  Captain  Robert  K.  McCoy  ( AFOTEC/Kirtland  Air  Force 
Base)  solicited  the  aid  of  the  SubTAG  for  inputs  regarding  his 
attempt  to  use  a  networking  model  approach  at  the  Consolidated 
Space  Operations  Center.  Mr.  Hartzell  noted  that  the  group 
discussed  the  possible  interaction  between  Captain  McCoy's 
approach  and  requirements  and  efforts  discussed  by  Mr.  McNeese. 

Mr.  Hartzell  reported  that  Dr.  Edwin  R.  Smootz  (ARI 
FU/Hood )  had  directed  the  SubTAG' s  attention  to  the  need  for 
improved  field-derived  data  since  these  data  are  being  utilized  as 
inputs  to  modeling.  Dr.  Norman  E.  Lane  (Human  Factors  Society 
Representative)  reminded  the  group  to  be  alert  to  changes  in 
goals,  noting  that  if  the  model  is  not  sensitive  to  goal  changes, 
it  will  lack  validity. 

Mr.  Hartzell  announced  that  the  25  attendees  had 
approved  the  Manned  System  Modeling  Charter. 

4.2.5  Sustained/Continuous  Operations  —  Lieutenant 
Colonel  Gerald  P.  Krueger,  Walter  Reed  Army  Institute  of  Research, 
Washington,  DC 
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See  Attachment  J  for  the  report  submitted  by  Lieutenant 
Colonel  Krueger. 

4.2.6  Tri-Service  Human  Factors  Standardization 
Steering  Committee  (HFSSC)  —  Mr.  Gerald  Chaikin,  US  Army  Missile 
Command,  Redstone  Arsenal,  Alabama 

View  Graphs  —  Attachment  K.l 

Minutes  of  7  May  1985  —  Attachment  K.2 

Mr.  Chaikin  presented  a  brief  overview  of  the 
Committee's  activities,  noting  that  the  preparation  of  the  DOD 
Human  Factors  Standardization  Program  (HFAC)  Plan  is  the 
subgroup's  primary  reason  for  existence.  The  HFAC  Plan  provides 
the  basis  for  efforts  in  the  human  factors  standardization  area 
for  a  five-year  period.  Inputs  to  the  document  are  essentially 
established  by  the  individuals  serving  on  HFSSC.  The  document  is 
revised  every  two  years  and  the  most  recent  revision  (Revision  4, 
dated  21  August  1985),  was  accepted  and  signed  as  submitted. 

Mr.  Chaikin  presented  status  reports  on  the  projects 
listed  below. 

.  HFAC-0009  (Manpower,  Personnel  and  Training  Data 

Requirements,  EPS).  John  L.  Miles,  Jr.  (ARI)  and 
Stephen  C.  Merriman  (DTDAC),  along  with  Air  Force 
participation  have  completed  the  final  report  of  this 
effort.  The  report  contains  specific  recommendations  on 
personnel  and  training  data  items,  including:  the 
review  of  99  data  items;  the  suggested  abolition  of  41 
data  items;  and  suggestions  for  future  efforts.  This 
report  will  be  circulated  in  November  1985  for  formal 
review  by  the  affected  agencies. 

HFAC-0016  (MIL-STD-1294A) .  This  effort  on  Acoustical 
Noise  Limits  in  Helicopters  has  been  completed; 
additional  data  items  were  included  in  the  document. 

.  HFAC-0024  (DOD-HDBK-761 ) .  This  completed  document  is  a 

reissuance  of  the  1983  (D.E.)  Hendricks  et  al.  Human 
Engineering  Guidelines  for  Management  Information 

Systems  in  DOD  Handbook  format.  The  UCI  SubTAG  will 
serve  as  the  agent  for  future  updates  of  this  document. 

.  HFAC-0027  (Human  Engineering  Procedures  Guide).  Boeing 

is  under  contract  to  prepare  this  document  irt1  DOD 
Handbook  format.  The  document  is  a  condensation  and 
update  of  previous  individual  Service  Guides  on  the 
imposition  and  implementation  of  MIL-H-46855  data  items. 
[For  the  status  of  other  HFSSC  projects,  seei"K.l-2.] 

In  closing,  Mr.  Chaikin  reported  that  the  SubTAG 
discussed  a  major  NASA  human  engineering  effort.  The  18-month 
Man-System  Integration  Standards  (MSIS)  project  began  in  July  1985 
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and  is  intended  to  develop  a  set  of  man-system  integration 
standards,  requirements,  and  guidelines  for  human  presence  in 
space.  MSISs  will  be  based  on  existing  standards,  requirements, 
and  guidelines  in  so  far  as  possible.  Discrepancies  and  voids  in 
existing  data  are  to  be  identified  and  resolved,  utilizing  the 
expertise  of  both  government  and  industry  personnel.  Products  of 
this  effort  are  expected  to  include: 

1)  a  volume  containing  criteria,  background,  data 
rationale  and  other  useful  information  for  designers  of  zero 
gravity  applications; 

2)  a  Standard,  similar  in  concept  to  1472; 

3)  a  pocket  edition  of  the  handbook,  similar  to  the 
Human  Engineering  Design  Data  Digest;  and 

4)  a  computerized  relational  database. 

The  HFSSC  has  formally  requested  that  the  Program  Manager  for  this 
effort  be  invited  to  present  at  the  next  TAG  plenary  session. 

4.2.7  Tri-Service  Workload  Coordinating  Committee  — 

Ms.  Heidi  M.  Fiedler,  Naval  Underwater  Systems  Center,  Newport, 
Rhode  Island 


See  Attachment  L  for  the  Report  and  visual 
materials  submitted  by  Ms.  Fiedler. 

4.2.8  User  Computer-Interaction  (Mr.  L.  Peterson)  — 

Dr.  John  J.  O'Hare,  Office  of  Naval  Research,  Washington,  DC 

View  Graphs  —  Attachments  M.l  through  M.7 

Dr.  O' Hare  reported  that  the  SubTAG  had  agreed  to  be  the 
agent  for  the  updating  of  DOD-HDBK-761A  and  for  the  preparation  of 
a  stand-alone  version  of  section  5.15  of  MIL-STD-1472C.  Schedules 
for  these  efforts  will  be  furnished  to  the  Tri-Service 
Standardization  Steering  Committee. 

In  other  action.  Dr.  O'Hare  noted  that  an  on-line 
database  of  annotated  documents  on  user-computer  interfaces  has 
been  established.  Access  to  a  demonstration  of  this  database, 
contracted  to  Battelle  Columbus  Laboratories,  can  be  obtained  by 
dialing  (614)  424-5850  [300  Baud  line]  and  (614)  424-5450  [1200 
Baud  line].  Plans  call  for  this  database  to  be  updated  on  a 
two-year  cycle.  [See  Attachment  M.7  for  complete  directions  to 
access  the  demonstration. ] 

The  SubTAG  has  initated  plans  for  an  electronic  bulletin 
board  and  has  scheduled  a  workshop  on  tools,  techniques,  and 
concepts  for  workstation  design.  The  latter  effort,  under 
STARS-funded  contract  to  Carlow  Associates,  is  scheduled  for  April 
1986  at  the  Naval  Postgraduate  School  in  Monterey,  California.  A 
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separate  UCI  Working  Group  meeting  in  Albuquerque,  New  Mexico,  is 
to  be  scheduled  once  plans  for  the  workshop  and  DOD  HFE  TAG 
Meeting  16  have  been  determined.  [The  Workshop  on  workstation 
design  has  been  scheduled  for  April  8-10  at  NPS.  No  date  has  been 
determined  for  the  Working  Group  meeting. ] 

Presentations  made  at  the  meeting  included: 

STARS.  Mr.  Larry  Peterson  (HEL)  outlined  the  Software 
Technology  for  Adaptable,  Reliable  Systems  Program,  which  evolved 
from  DOD  Mission-Critical  Computer  Resources  (MCCR)  needs.  The 
mission  of  the  multi-year,  multi-dollar  STARS  program  is  to 
enhance  software  usability,  maintainability  and  productivity  by 
improving  the  state-of-practice  in  the  acquisition,  development 
and  support  of  software  for  MCCRs.  General  objectives  of  the 
program  are  the  development  of  operable  workstations,  enhanced 
training  practices,  standardized  procedures,  and  accessible 
databases.  [See  Attachment  M.2.] 

USER,  System  and  Environment  Scaling.  Mr.  David  R.  Eike 
(Carlow  Associates)  described  a  STARS-funded  effort  relating  to 
the  development  and  validation  of  measures,  methods,  principles 
and  criteria  for  scaling  user  characteristics,  system  usability 
and  environmental  complexity.  The  goal  of  this  effort  is  to 
reduce  complexity.  [See  Attachment  M.3.] 

User-Workstation  Interface  Requirements.  Mr.  David  R. 
Eike  also  described  a  STARS-funded  effort  whose  objective  is  to 
develop  and  validate  a  comprehensive  set  of  user-workstation 
interface  design  requirements  and  tools  which  could  be  utilized  by 
a  variety  of  users,  from  managers  to  programmers.  [See  Attachment 
M.4.  ] 

UCI/STARS  Workstation  Interoperability  Consortium.  Dr. 
Mark  M.  Brauer  (Lockheed  California  Corporation)  reported  on  his 
efforts  to  organize  the  21  member  consortium.  The  consortium, 
composed  of  key  industry  and  technical  society  professionals  in 
the  UCI  domain,  has  volunteered  its  time  to  support  the  overall 
objectives  of  the  UCI  SubTAG;  and  is  intended  to  provide  a  forum 
for  UCI  information  exchange  among  government,  technical  societies 
and  industry  groups.  The  consortium  will  explore  and  define 
common  areas  of  need,  UCI  tools,  methods  and  references;  and  will 
develop  recommendations  for  consideration  by  the  UCI  SubTAG  in  the 
areas  of  UCI  standards,  guidelines,  tools,  methods  and  research. 
[See  Attachment  M.5.] 

COPE.  Mr.  Michael  D.  McNeese  (AAMRL)  gave  an ‘overview 
of  C 3  Operator  Performance  Engineering  technology,  describing 
user-computer  interaction  issues  relevant  to  integrated  analyses 
techniques  for  C  ^  systems,  adaptive  aiding  and  integrated  C7 
workstation  design.  y 

Mr.  McNeese  also  gave  a  brief  overview  of  the  Rapid 
Intelligent  Prototyping  Laboratory  (RIPL)  program.  [See 
Attachment  M.6  for  materials  relating  to  these  two  programs.] 
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4.2.9  Voice-Interactive  Systems  —  Mr.  Clayton  R. 

Coler,  NASA-Ames  Research  Center,  Moffett  Field,  California 

Agenda  and  Attendee  List  —  Attachment  N 

Mr.  Coler  reported  that  in  addition  to  the  meeting 
held  on  5  November  the  group  has  scheduled  a  tour  of  the  Naval 
Ocean  Systems  Center  (NOSC)  and  ITT  facilities  and  an  additional 
technical  session  following  the  conclusion  of  the  TAG.  Minutes  of 
both  technical  sessions  will  be  sent  directly  to  Voice  SubTAG 
attendees . 


Presentations  at  the  5  November  session  included: 

NOSC's  Role  in  Speech  Recognition  and  Synthesis 
(Ms.  Elaine  Schiller ) .  Current  Voice  Input/Output  ( I/O ) 
activities  at  NOSC  cover  a  wide  range  of  applications  including 
voice-entry  to  training  devices;  voice-interactive  expert  systems; 
and  natural  language  understanding.  Future  work  at  NOSC  is 
planned  to  emphasize  Voice  I/O  CJ  operations  and  standardization 
of  interfaces,  data  busses  and  Voice  I/O  hardware  and  software. 

An  Architecture  for  Parallel  Proessing  in  Speech 
Recognition:  The  Trace  Model  (Dr.  Jeffrey  Ellman,  University  of 

California  at  San  Diego).  Dr.  Ellman  discussed  current  machine 
speech  recognition  limitations  compared  to  the  human  ability  to 
understand  speech.  Current  machine  systems  tend  to  process 
information  in  a  serial  fashion,  whereas  humans  are  able  to 
integrate  a  large  amount  of  information  over  a  short  period  of 
time.  Dr.  Ellman  proposed  a  network  system  which,  although  not 
intending  to  simulate  the  function  of  the  human  brain,  serves  to 
produce  results  consistent  with  the  results  seen  when  humans 
process  and  understand  speech.  Focus  will  be  on  the  integration 
of  large  amounts  of  information  and,  as  recognition  proceeds, 
checks  will  be  made  to  determine  if  the  recognition  is  tracking  in 
a  meaningful  manner. 

Interference  Reduction  (Mr.  Stephen  Smith,  Rome  Air 
Development  Center).  This  presentation  was  an  overview  of  the 
efforts  at  RADC  in  the  development  of  a  speech  enhancement  unit 
that  cancels,  in  real  time,  three  kinds  of  noise:  impulse  noise, 
narrow-band  noise,  and  wide-band  noise.  The  system  currently  is  a 
large  unit  and  is  not  so  usable  in  severe  environments.  A 
contract  has  been  awarded  to  fabricate  a  prototype  unit  that  will 
be  flyable  in  aircraft  and  usable  in  field  situations.  The 
prototype  is  expected  to  be  available  in  March  1986. 

Presentations  scheduled  for  8  November  include: 

1 .  Voice  Interactive  Avionics  (VIA)  Flight  Test 
Preparations ,  Mr.  Lockwood  Reed ,  USA  Avionics  R&D  Activity,  Fort . 
Monmouth,  New  Jersey; 
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2 .  Software  Methodology  for  Automated  Recognition 
Training  (SMART),  Mr.  David  Williamson,  AFWAL/FIGR, 
Wright-Patterson  Air  Force  Base,  Ohio; 

3 .  Voice  Research  at  the  Human  Engineering 
Laboratory,  Ms.  Kathy  Christ,  USA  Human  Engineering  Laboratory, 
Aberdeen  Proving  Ground,  Maryland; 

4 .  Application  of  Connected  Speech  Recognition  to 
Tactical  Voice  Command  and  Speaker  Authentication,  R.  Wohlford  and 
B.  P.  Landell,  ITT  Defense  Communications  Division,  San  Diego, 
California;  and 

5.  DARPA  Speech  Research,  Dr.  Tice  DeYoung, 
SPAWARSYSCOM,  Washington,  DC. 

During  the  business  session,  new  Roadmap  forms  were 
distributed.  Roadmaps  are  summary  written  descriptions  of  Voice 
I/O  projects.  They  document  work  in  progress,  including 
objectives,  approach,  results  to  date,  and  specific  facilities  and 
equipments  being  utilized.  Roadmaps  serve  to  inform  members  of 
the  Voice  SubTAG  about  ongoing  work  in  other  laboratories  and 
stimulate  more  detailed  information  exchange  between  centers  who 
have  related  work  in  progress. 

In  other  business,  Mr.  Coler  announced  that  Mr.  Joe 
Pete  Severo  (ARI)  has  volunteered  to  provide  the  Voice  subgroup 
with  a  document  concerning  the  use  of  ARPANET.  In  the  past,  the 
ARPANET  has  served  the  SubTAG  effectively  by  providing  a  means  of 
information  regarding  new  work  efforts,  new  reports,  and  updated 
technical  information.  ARI  will  serve  as  the  host  of  the  new 
Voice  ARPANET  mailbox. 

Mr.  Coler  reported  that  the  SubTAG  has  undertaken 
the  development  of  a  User's  Guide  to  Voice  I/O  Systems.  The  guide 
in  intended  to  provide  specific  Voice  I/O  System  information  for 
new  users.  Mr.  David  Williamson  (AFWAL)  is  in  charge  of  this 
effort . 


The  group  also  discussed  the  need  to  document 
current  state-of-the-art  capabilities  of  Voice  I/O  systems  and  the 
identification  of  critical  knowledge  gaps,  where  needed 
information  is  not  expected  to  be  available  in  a  timely  fashion. 
This  effort  would  serve  to  identify  candidate  areas  for  future 
research  and  development  funding. 

In  closing,  Mr.  Coler  expressed  his  appreciation  to 
NOSC  for  hosting  the  8  November  technical  session  and  to  Ms. 

Elaine  Schiller  and  Mr.  Stephen  Nunn  for  arranging  the  NOSC  tour. 
Mr.  Coler  also  reported  that  Ms.  Schiller  has  been  selected  as  the 
new  Voice-Interactive  Systems  Chair.  fj 
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5.  PROFESSIONAL  AND  TECHNICAL  PRESENTATIONS 

5 . 1  Tutorial:  Selecting  Performance  Measures  in  Human 
Factors  Engineering  Testing  and  Research 

Moderator  —  John  L.  Miles,  Jr.,  J.D.  (ARI) 

Panelists 

Army  —  James  C.  Geddie,  Ph.D  ( TCATA/HEL ) 

Navy  —  Dennis  K.  McBride,  Ph.D  (PMTC) 

Air  Force  —  Stephen  M.  Rokicki ,  Ph.D  (OTEC) 

APA/21  —  Frederick  A.  Muckier,  Ph.D  (APA/21 
Representative ) 

View  Graphs  —  Attachments  P  through  U 

Dr.  Miles  introduced  the  panel  members  and  gave  a  brief 
overview  of  the  format  and  intent  of  the  presentation.  Dr.  Miles 
noted  that  although  the  briefings  were  slanted  toward  those 
individuals  directly  involved  in  performance  measurement,  an 
awareness  of  the  underlying  problems  would  be  of  benefit  to  all 
attendees.  According  to  Dr.  Miles,  no  definition  of  a  performance 
measure  can  be  found  in  relevant  dictionaries  but  the  purpose  of  a 
performance  measure  is  to  select  from  the  many  facets  of 
observable  activity  one  (or  more)  which  is  assessable  in  relevant, 
meaningful  quantifiable  units.  Dr.  Miles  suggested  four  reasons 
why  performance  measures  are  important: 

1)  validity  of  human  factors  technology; 

2)  communication  of  findings  in  units  that  have  meaning  for 
the  user; 

3)  construction  of  databases;  and 

4)  uses  of  new  technology. 

Dr.  Miles  reiterated  that,  at  present,  there  is  no  one,  best, 
universal  performance  measure.  The  most  commonly  used  human 
performance  measures  are  time  and  "accuracy,"  where  accuracy  is 
assumed  to  be  a  reciprocal  of  the  measred  error.  There  have  been 
problems  with  this  approach,  necessitating  a  MANPRINT  rule  that 
says  that  time  and  accuracy  are  not  independent  of  each  other. 
Common  problems  with  performance  measures  are  that  they  are 
frequently  set  too  broadly  or  too  narrowly,  they  are  often 
unrelated  to  military  concerns  and  perhaps  most  critical,  they 
often  fail  to  discriminate. 

According  to  Dr.  Miles,  the  ideal  source  for  performance 
measures  would  be  the  requirements  document  for  the  system;  and 
ideally  the  requirements  document  would  include  operator 
performance  standards,  operator  characteristics ,  and  information 
regarding  the  maximum  tolerable  training  burden  (cost,  length, 
etc. ) . 
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Human  Performance  Measurement:  Context  and  Content  —  Dr.  Geddie 

For  systems  recently  fielded  by  the  Army  and  for  those  still 
in  development,  it  is  typical  for  the  life  cycle  cost  projections 
for  a  system  to  show  only  about  20%  of  the  total  cost  for  the 
system  to  be  absorbed  by  research,  development  and  acquisition 
costs.  The  remaining  80%  is  absorbed  by  operation,  maintenance 
and  other  support  costs.  This  later  80%  is  heavily  weighted  with 
"people  costs"  to  include  trainers,  operators,  maintainers  and 
others.  Thus  it  has  become  increasingly  apparent  to  Army  planners 
that  greater  emphasis  on  developing  systems  which  have  an 
effective  and  efficient  soldier-machine  interface  —  even  at  the 
expense  of  increasing  the  up-front  acquisition  costs  —  promises 
soldier-machine  systems  whose  total  life  cycle  cost  effectiveness 
is  dramatically  enhanced. 

Soldier-Machine  Systems  vs  Hardware 

This  increased  emphasis  on  the  soldier-machine  interface  has 
not  been  a  sudden  change.  Instead  it  has  been  a  gradual  one 
coincident  with  and  related  to  an  increased  willingness  to  think 
in  terms  of  developing  total  systems  rather  than  developing 
separate  hardware,  training,  software,  logistic  support,  technical 
documentation  and  facilities  as  separate  efforts  to  be  later 
combined  into  an  effective  system.  It  should  be  added  at  this 
point  that  the  metamorphosis  is  by  no  means  complete.  There  are 
many  members  of  the  acquisition  community  who  still  say  "system" 
when  what  they  really  mean  is  "hardware". 

At  first  encounter  the  difference  between  the  two  concepts 
may  seem  somewhat  superficial,  but  it  has  fundamental  implications 
for  the  way  in  which  the  Army  develops  new  combat  systems.  It 
also  has  some  profound  consequences  for  testing  and  evaluating 
developmental  systems. 

During  development,  the  systems  approach  requires  that  many 
players  get  into  the  process  right  at  the  beginning.  The  materiel 
developer,  the  combat  developer,  the  trainer,  the  logistician  and 
several  others  must  participate  in  defining  goals,  requirements 
and  limitations  for  the  system.  Unlike  the  somewhat  fragmented 
approach  to  development  mentioned  above,  the  systems  approach  re¬ 
quires  that  each  player  participate  in  a  trade-off  process.  In 
this  process,  an  attempt  is  made  to  arrive  at  a  cost-effective 
means  of  acquiring  a  new  combat  capability.  Among  the  trade-offs 
which  might,  for  example,  be  negotiated  is  a  choice  between  (1)  a 
hardware  design  which  is  high  in  acquisition  costs  but  -imposes  hu¬ 
man  performance  and  skill  requirements  which  are  cheap -to  acquire 
and  maintain  versus  (2)  a  hardware  design  which  is  lower  in  acqui¬ 
sition  costs,  but  requires  human  performance  and  skills  which  may 
be  very  costly  to  acquire  and  maintain.  Based  upon  the  available 
cost  predictions,  the  alternative  meeting  the  system  goals  which 
offered  the  lowest  life  cycle  cost  could  be  selected.  Now,  what 
does  all  of  this  have  to  do  with  human  performance  measurement? 

*  Text  provide  by  Dr.  Geddie. 
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Implication  for  Performance  Measurement 


Once  the  focus  of  attention  shifts  from  "materiel"  to 
"system",  the  business  of  testing  and  evaluation  becomes  quite 
different  from  testing  of  materiel  alone.  In  component  level  and 
even  subsystem  level  testing,  hardware  functions  are  exercised  in 
a  way  in  which  any  human  function  is  assumed  to  have  a  probability 
of  1.0  of  being  correctly  performed  the  first  time  and  every  time, 
regardless  of  the  conditions  surrounding  the  performance.  The 
concern  in  this  type  of  testing  is  nothing  more  than  "Did  the 
hardware  behave  as  predicted?".  The  assumption  is  explicit  that 
failures  due  to  human  error  are  not  chargeable  to  the  hardware 
design.  An  implicit  (and  probably  wrong)  assumption  of  this 
approach  to  development  and  testing  is  that  the  combination  of 
manpower  and  training  resources  available  to  the  Army  when  this 
hardware  is  fielded  will  be  capable  of  meeting  whatever  human 
performance  requirements  have  been  built  into  the  system. 

Testing  and  evaluation  conducted  under  a  systems  approach 
acknowledges  the  influence  of  the  human  operator  and  his  or  her 
performance  on  total  system  effectiveness  and  reliability. 
Soldier-in-the-loop  testing  attempts  to  exercise  the  system  using 
a  sample  of  soldiers  who  by  aptitude,  training,  experience,  and 
physical  characteristics  are  typical  of  those  who  will  function  as 
part  of  the  system  once  it  is  fielded.  An  attempt  is  made  to 
control  the  variation  in  this  soldier  sample  by  use  of  seletion 
criteria  or  at  least  to  account  for  it  with  demographic  data. 
Within  available  test  resources,  system  functions  are  exercised 
over  a  representative  sample  of  the  conditions  (terrain,  weather, 
visibility,  etc. )  which  are  anticipated  in  use  of  the  system  in 
training  and  in  combat. 

In  the  interest  of  efficient  use  of  test  resources,  human 
factors  testing  is  frequently  done  concurrently  with  testing  for 
other  aspects  of  system  performance.  For  example,  in  testing  a 
vehicular  system,  data  on  critical  driving  tasks  might  be 
collected  on  the  same  exercises  used  to  test  reliability  or 
durability  of  the  vehicle. 

Human  Factors  Test  Data 


Whether  human  factors  data  are  collected  in  a  separate  test 
or  during  testing  of  some  other  aspect  of  performance,  there  are 
basically  three  kinds  of  information  which  will  be  collected. 

These  are  (1)  engineering  measurements,  (2)  user  opinion,  and  (3) 
human  performance  data.  Each  has  its  own  appropriate  role  in  the 
evaluation  and  will  be  treated  separately. 

Engineering  Measurements.  When  the  project  management  office 
or  the  prime  contractor  on  a  given  system  begins  to  plan  for  human 
factors  testing,  engineering  measurements  usually  are  considered 
first.  During  establishment  of  the  system  goals,  requirements  and 
limitations,  applicable  sections  of  military  standards  and 
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specifications  should  have  been  cited  as  design  standards  to  be 
met.  The  requirements  document  may  also  specify  military 
handbooks  and  other  sources  of  guidance  to  be  used  in  hardware 
design.  In  general,  engineering  measurements  are  the  data  used  to 
evaluate  compliance  with  these  requirements.  The  required  data 
may  often  be  collected  without  participation  of  the  human 
component  of  the  system  and  may  address  requirements  in  such  areas 
as : 


size 

weight 

lighting  level 

noise  level 

crew  workspace  layout 

ingress  and  egress  provisions 

temperature 

vibration 

display  brightness,  legibility  and  labeling 
control  placement  and  force  requirements. 

User  Opinion.  This  second  category  of  information  collected 
for  a  human  factors  evaluation  is  used  to  learn  from  test 
participants,  test  control  personnel  and  observers  about 
characteristics  of  the  system  which  might  not  be  revealed  by 
engineering  measurements.  It  is  generally  accepted  that  troops 
will  not  function  at  optimum  effectiveness  with  equipment  they 
dislike  or  mistrust,  thus  limiting  system  performance.  Obviously, 
the  most  direct  way  to  find  out  how  a  user  feels  about  the 
equipment  is  to  ask  his  opinion.  The  user's  insights  —  after  he 
has  attempted  to  function  with  the  hardware  and  procedures  as  part 
of  a  total  system  —  also  can  sometimes  suggest  improvements  in 
hardware  design  or  in  operating  procedures  which  would  not  be 
revealed  by  checking  for  compliance  with  specifications  and 
standards.  Problems  reported  by  users  may  also  identify  system 
characteristics  which  merit  close  scrutiny  during  human 
performance  testing. 

Human  Performance  Data.  The  overall  goal  of  a  human  factors 
engineering  program  is  to  insure  compatibility  among  (1)  the 
soldier,  (2)  the  training,  (3)  the  tasks,  and  (4)  the  equipment. 
Having  compared  the  hardware  characteristics  against  human 
engineering  standards  and  trained  a  sample  of  soldiers  and 
elicited  opinions  of  the  equipment  and  training,  the  remaining 
task  in  determing  whether  the  human  factors  engineering  program 
has  met  its  goal  is  exercising  mission-critical  soldier  tasks  and 
collecting  and  analyzing  performance  data  on  those  task^. 

Tp 

Collection  of  these  data  must  be  based  on  a  review  of  the 
human  performance  requirements  associated  with  the  system.  If  the 
performance  requirements  have  been  properly  specified  in  the 
requirements  documents  and  a  useable  task  analysis  is  available,  a 
good  start  has  already  been  made  on  identifying  tasks  on  which  to 
collect  human  performance  data.  Other  inputs  to  the  task 
selection  process  should  include  technical  manuals  and  training 
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materials  used  in  training  test  participants.  The  list  of  tasks 
to  be  included  should  have  as  its  highest  priority  those  tasks 
whose  performance  defines  an  outer  limit  on  total  system 
performance  (e.g.,  in  a  tank  system:  loading,  laying  and  firing 
the  gun ) . 

There  are  two  basic  measures  used  in  human  performance 
measurement:  human  performance  time  and  error  rate.  For  each 

task  exercised  and  measured  in  the  test  both  kinds  of  data  must  be 
collected  on  each  event.  The  reason  for  insisting  that  both 
measures  be  made  of  each  event  is  that  for  most  tasks,  performance 
time  and  error  rate  can  be  traded  off  one  for  the  other.  The  set 
established  in  the  test  participant  can  radically  affect  whether 
he  or  she  emphasizes  speed  of  performance  at  the  expense  of 
accuracy  or  conversely,  accuracy  at  the  expense  of  speed.  For 
some  tasks,  the  trade-off  function  itself  may  be  more  important 
than  either  data  point  alone  in  affecting  both  design  changes  and 
operational  doctrine  for  the  system. 

Analysis  of  these  data  should  first  compare  achieved 
performance  against  the  performance  goals  established  in  the 
system  requirements  documents.  If  the  system's  front-end  analyses 
have  been  thorough  enough  to  define  criteria  for  task  performance, 
they  define  the  test  criteria.  In  the  case  of  systems  for  which 
the  criteria  have  not  been  stated,  the  performance  data  are  used 
in  predicting  what  the  performance  in  the  field  will  be.  The 
question  of  "How  good  is  good  enough?"  then  gets  a  post  hoc 
answer,  but  at  least  the  decision  of  whether  to  accept  a  system 
can  be  an  informed  one,  based  upon  knowledge  about  currently 
achieved  performance. 

Another  use  —  perhaps  a  more  important  one  —  for  these  data 
is  identifying  areas  in  which  human  factors  engineering  design 
improvements  have  a  high  potential  payoff.  If,  for  example,  the 
data  show  an  unexpectedly  long  performance  time  for  one  of  a 
series  of  sequentially  performed  tasks,  that  task  would  be 
identified  as  a  priority  candidate  for  improvement  in  the  hardware 
associated  with  it  or  the  procedures  for  performing  it. 
Consideration  might  also  be  given  to  machine  aiding  or  to 
automating  part  or  all  of  its  performance. 

Soldier-Machine  System  Evaluation 

The  bottom  line  for  evaluation  of  a  soldier-machine  system 
has  been  reached  when  the  evaluator  has  answered  the  question,  "So 
what?",  for  each  test  issue.  The  specifications  and  standards 
against  which  we  evaluate  engineering  measurements  should  be  met. 
Their  criteria  have  been  developed  from  experience  with  many  past 
systems  and  meeting  those  criteria  improves  the  probability  of 
acquiring  an  effective  and  efficient  soldier-machine  system. 
However,  meeting  those  requirements  does  not  insure  that  this  has 
been  achieved,  nor  does  failure  to  meet  one  or  more  of  the 
criteria  insure  an  unsuccessful  system. 
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Likewise,  the  user's  feelings  and  attitudes  toward  the  system 
are  important.  The  Army  has  demonstrated  too  many  times  that  a 
system  that  is  not  liked  or  trusted  by  the  users  has  little  chance 
of  operational  success.  However,  it  is  critically  important  to 
keep  in  mind  that  there  is  not  a  one  for  one  correspondence 
between  equipment  characteristics  about  which  users  complain  and 
equipment  characteristics  which  can  be  shown  to  degrade  their 
performance  or  total  system  performance.  In  competitive  testing 
it  has  also  been  observed  that  soldiers  often  state  a  preference 
for  an  equipment  design  with  which  their  performance  is  worse  than 
with  a  less  liked  competitor.  The  lesson  we  must  acknowledge  from 
that  reversal  is  that  the  user  can  give  us  his  feelings  about  how 
well  he  likes  a  system,  and  is  seldom  reluctant  to  evaluate  its 
and  his  performance,  but  regardless  of  how  precisely  and 
accurately  we  may  measure  those  feelings  and  attitudes,  they  have 
(at  least  in  my  small  sample  of  opportunities  to  compare  them  with 
objective  performance  data)  been  observed  to  have  no  correlation 
with  objectively  measured  performance. 

The  indications  from  engineering  measurements  and  from  user 
input  are  important  in  a  system  evaluation.  They  should  influence 
the  selection  of  tasks  for  human  performance  testing.  Obviously, 
if  an  equipment  characteristic  clearly  violates  a  human 
engineering  standard  or  users  feel  that  it  significantly  degrades 
their  performance,  that  characteristic  merits  closer  examination. 
But  both  the  system  and  the  human  fators  profession  and  technical 
area  are  done  a  disservice  when  we  attempt  to  evaluate  system 
performance  or  human  performance  as  one  of  its  components  without 
measuring  the  performance.  The  risk  to  the  system  is  simply  that 
our  evaluation  may  be  wrong  and  we  end  up  "fixing  something  that 
isn't  broken"  or  accepting  a  system  with  design  errors  that  reduce 
performance  without  our  even  knowing  that  performance  has  been 
reduced.  The  long-term  damage  to  human  factors  may  be  much  more 
significant.  Human  factors  as  a  discipline  is  continually 
challenged  to  document  its  value  to  system  design.  When  we 
justify  decisions  to  change  (or  not  to  change)  a  system's  design 
upon  a  prediction  of  performance  consequences,  then  base  those 
predictions  upon  anything  less  than  hard  performance  data,  both 
our  credibility  and  future  acceptance  of  human  engineering  inputs 
are  jeopardized. 

Put  a  bit  more  colloquially, 

if  we  want  to  know  whether  the  materiel  design 
has  complied  with  the  system  specification  and 
with  applicable  criteria  in  the  standards, 
engineering  measurements  are  appropriate;  if 
we  want  to  know  whether  the  user  likes  it, 
user  opinion  is  appropriate;  but  if  we  want  to 
know  how  well  the  humans  in  the  system 
perform,  and  how  their  performance  relates  to 
total  system  performance,  there  is  no 
defensible  substitute  for  human  performance 
measurement. 

U 
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The  Navy  Experience  --  Dr.  McBride 

Dr.  McBride  reported  that  there  are  a  number  of  areas  on 
which  aircrew  performance  measurement  impacts,  including  the 
mathematical  modeling  of  pilot  behavior,  workload  evaluation, 
aircrew  training,  evaluation  of  aircrew  perceptions  and  decisions, 
etc.  A  typical  example  of  performance  measurement  in  the  aircrew 
environment  might  be  that  of  the  measurement  of  deviation  from  a 
prescribed  course  of  actions.  Dr.  McBride  noted  that  most  of  the 
processes  utilized  in  flying  aircraft  are  available  for  our 
examination  and  measurement,  whether  they  be  perceptual,  motor  or 
physiological.  The  aircraft  also  provides  outputs  from  aileron 
movement,  attitude  direction  indicator,  etc.  and  from  1553  data 
thruput.  In  terms  of  man-machine  output,  other  specific  measures 
could  be  aceleration,  altitude,  airspeed,  error  measurements,  etc. 
Other  measurements  from  this  high  workload  environment  include 
frequency  measurements  such  as  number  of  control  losses,  number  of 
control  reversals,  number  of  correct  responses. 

There  are  also  some  combined  overall  measures  available  -- 
grid  stick  index,  landing  performance  score,  objective  mission 
success  scores,  trials  to  criterion.  According  to  Dr.  McBride, 
the  range  of  measurements  is  wide  —  from  use  of  "rule  of  thumb" 
measurements  through  to  sophisticated  measures  such  as  the 
Dopple-measured  blood  velocity  at  the  level  of  the  brain. 

Dr.  McBride  concluded  his  presentation  by  reminding  the  TAG 
that  there  are  a  number  of  parameters  of  measurement  that  must  be 
considered,  including  fidelity  of  test  conditions,  cost  of  the 
measuremetn  system,  ease  and  flexibility  of  testing,  control,  and 
data  accessibility.  Since  no  one  measure  is  high  on  each  of  these 
scales,  it  is  necessary  to  titrate  and  use  the  scales  as  best  one 
can.  For  additional  information  on  performance  measures.  Dr. 
McBride  recommended  that  interested  individuals  obtain  a  copy  of 
the  January  1982  Naval  Training  Equipment  Center  Technical  Report 
IH-330  by  T.  R.  Mixon  and  W.  F.  Moroney  entitled  An  Annotated 
Bibliography  of  Objective  Pilot  Performance  Measures. 

Human  Performances  Evaluation  in  Air  Force  Operational  Test 
and  Evaluation  —  Dr.  Rokicki 

Dr.  Rokicki  reported  that  the  current  approach  to  human  fac¬ 
tors  evaluations  at  the  AF  Operational  Test  and  Evaluation  Center 
is  that  it  may  not  be  necessary  to  do  complete  human  factors  test¬ 
ing  on  every  system:  that  in  some  systems  human  performance  may 
not  be  critical.  Since  hardware,  software,  human  performance  and 
the  mission  comprise  total  system  performance,  there  are  cases  in 
which  it  is  not  necessary  to  have  optimal  human  performance:  if 
the  system  performs  well,  then  effective  and  acceptable  human 
performance  may  be  all  that  is  required. 

Dr.  Rokicki  noted  that  Air  Force  operational  test  and 
evaluation  concentrates  on  two  principal  areas.  One  area  is  the 
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dynamic  approach  to  the  man-system  and  group  interaction,  where 
the  major  questions  are  "does  the  system  constrain/prevent  the 
human  from  doing  his  job?"  and  "does  the  human  hinder  the 
performance  of  the  system?"  The  other  area  of  concentration  is  a 
focus  on  intrinsic  human  performance  in  terms  of  physiological  and 
psychological  decrement  and  fatigue/workload  issues.  Evaluation 
techniques  include: 

.  workload  analysis 

.  facility  and  workstation  evaluation 

.  error  rates 

.  display-task  analysis 

.  environmental  evaluation 

.  small  group  analysis 

.  behavioral  analysis. 

Workload  Analysis.  Since  high  workload  degrades  operator 
performance,  OT&E  utilizes  a  variety  of  workload  analysis 
techniques,  including  the  Subjective  Workload  Assessment  Technique 
(SWAT),  the  modified  Cooper-Harper  Scale,  physiological  correlates 
and  other  objective  and  subjective  measurements. 

Facility  and  Workstation  Evaluation.  To  determine  if  the 
design/configuration  of  a  system  prevents/hinders/limits  operator 
performance,  operational  evaluations  are  made  of  the  system  layout 
and  console  layout.  These  evaluations  are  intended  to  provide  a 
dependable  base  to  assess  operational  impacts. 

Display-Task  Analysis.  These  analyses  are  performed  to 
determine  if  the  display  formats  and  presentation  sequences  are 
adequate  for  the  operational  tasks.  Technical  evaluations  of  the 
formats  and  controls,  observations,  subjective  reports  and  video 
tapes  are  part  of  this  type  of  analysis. 

Environmental  Evaluation.  In  the  Air  Force,  environmental 
evaluations  are  normally  done  during  developmental  test  and 
evaluation  (DT&E).  However,  conditions  under  which  DT&E  is  done 
may  not  always  reflect  operational  conditions;  and  acoustics, 
lighting,  temperature  control,  humidity,  etc.  pan  affept  operator 
performance  as  well  as  the  hardware/software  components  of  the 
system. 

Behavioral  Analysis.  One  of  the  techniques  used  by  the 
Center  is  that  of  analyzing  the  cause  and  effect  of  critical 
incidents  involving  operator  fatigue,  workload,  stress,  etc. 

These  postmortem  analyses  are  only  done  if  the  critical  incident 
adversely  affected  system  performance.  According  to  Dp.  Rokickp, 
this  focus  saves  time,  money,  and  manpower. 

-.1 

In  closing.  Dr.  Rokicki  reiterated  that  although  there  are 
specifications  and  standards  for  other  system  components,  there 
are  no  specific  task  standards  nor  parameters  for  human 
performance;  and  at  present,  there  is  no  universally  accepted 
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human  performance  evaluation  methodology.  These  issues  need  to  be 
addressed  but  Dr.  Rokicki  suggested  using  caution  when  developing 
standards  for  human  performance  due  to  the  tremendous  variations 
in  human  behavior. 

Selecting  Performance  Measures:  Some  Practice  and  Some 

Theory  --  Dr.  Muckier 

Over  the  last  25  years.  Dr.  Muckier  has  participated  in  a 
number  of  developmental  and  operational  tests  of  man-machine 
systems  and  it  has  been  his  experience  that  in  each  of  the  tests 
some  aspect  of  human  performance  measurement  has  been  a  serious 
problem.  Focusing  on  22  operational  system  tests  in  which  he  was 
involved  in  the  last  three  years.  Dr.  Muckier  noted  some  general 
characteristics  of  these  tests.  According  to  Dr.  Muckier,  each 
testing  situation  represents  a  new  configuration  —  calling  for 
judgements  as  to  which  measurements  will  be  of  the  most  value. 

The  high  cost  of  tests  and  the  severe  time  constraints  posed 
impact  both  the  test  plan  and  conduct.  Users  of  the  test  data 
often  prefer  simple  measurements;  and  since  there  are  differing 
viewpoints  as  to  purpose  of  the  testing,  it  can  be  difficult  to 
reconcile  evaluative  versus  diagnostic  approaches.  Other 
constraints  concern  a  particular  measurement's  "acceptability": 
i.e.,  is  the  measurement  the  appropriate  one  or  appear  to  be  the 
appropriate  one  for  the  parameter (s)  being  assessed?  Problems 
also  arise  due  to  the  great  differences  in  the  experience  and 
training  levels  of  the  testers,  evaluators  and  those  responsible 
for  the  system  being  tested.  Although  there  is  a  strong  demand 
for  "expert  judgements,"  there  is  also  a  marked  resistence  to 
human  performance  measurement. 

Instrumentation  can  also  contribute  to  test  difficulties: 
not  all  facilities/situations  have  adequate  instrumentation  for 
the  task(s)  at  hand  and  it  may  be  possible  that  what  ends  up  being 
measured  is  directly  related  to  how  much  instrumentation  can  be 
made  available.  On  the  other  end  of  the  instrumentation 
availability  issue  is  the  situation  in  which  a  complete,  automated 
computer-based  performance  measurement  system  is  utilized.  In 
this  instance,  the  sheer  amount  of  data  collected  may  result  in  an 
"overkill"  situation  in  which  there  are  too  many  data  points  to 
analyze  in  a  timely,  meaningful  way. 

Dr.  Muckier  indicated  that  these  problems  will  continue 
unless  improvements  are  made  in  human  performance  measurement. 
While  there  are  some  situations  in  which  quantitative  models  are 
available  and  can  be  easily  utilized,  in  most  cases  the  test  plans 
are  the  result  of  the  test  team's  personal  methodology. 

Dr.  Muckier  posed  a  model  for  performance  measurement,  noting 
that  the  basic  question  underlying  any  measurement  should  be  "what 
do  I  need  to  know?"  The  first  step  in  Dr.  Muckler's  model  is  an 
information  needs  analysis.  The  second  step  is  a  performance 
objectives  analysis  and  deals  with  the  issue  of  defining  standards 
for  performance.  The  third  step  is  an  analytic  process  in  which 
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one  separates  out  the  necessary  and  sufficient  set  of  performance 
parameters  (state  system  identification).  The  information  gleaned 
in  these  first  three  steps  is  integrated;  and  in  step  four, 
measurement  dimensions  are  specified.  Dr.  Muckier  noted  that 
these  first  four  steps  are  the  "what"  steps  of  the  model  and 
initially,  should  be  kept  separate  from  the  "how"  steps. 

Once  one  has  determined  what  he  needs  to  know,  the  next  step 
is  how  to  get  the  necessary  data  and  since  there  are  alternative 
measure  sets  for  most  of  the  measurement  dimensions  determined  in 
step  four,  it  becomes  necessary  to  compare  these  alternatives. 
According  to  Dr.  Muckier,  potential  criteria  for  determining  which 
measurement  set  to  use  include: 

.  validity 

.  reliability 

.  generalizability 

.  utility  (cost) 

.  precision 

.  nonreactivity 

.  acceptability. 

Dr.  Muckier  noted  that  not  all  test  personnel  are  acquainted  with 
the  concepts  of  validity  and  reliability.  It  is  also  necessary  to 
determine  the  level  of  precision  required;  if  this  is  not  done, 
the  quantity  or  quality  of  the  data  may  render  it  unusable.  It  is 
also  necessary  to  guard  against  having  the  process  of  measuring 
interfere  with  the  parameter  being  measured:  some  measurements 
are  more  reactive  (intrusive)  than  others.  Two  aspects  of 
generalizability  should  be  considered:  1)  internally,  has  the 
subset  been  selected  that  allows  generalizations  and  2)  can 
generalizations  about  the  data  be  made  across  all  classes  of 
systems.  Other  criteria  for  selecting  a  measurement  set  include 
costs  involved  in  the  data  collection  (instrumentation,  personnel, 
time,  etc. )  and  whether  the  measurement  set  is  acceptable  to  those 
responsible  for  the  system  being  tested. 

Step  six  is  the  preliminary  testing  of  alternative  measures 
and  Dr.  Muckier  indicated  that  it  is  not  always  possible  to  do 
this  stage.  When  the  opportunity  exists,  however,  three  aspects 
should  be  considered:  the  resources  available/necessary  to 
conduct  a  quality  test,  the  instrumentation  and  data  processing 
requirements,  and  methods  of  checking  the  quality  of  the  data 
during  the  measurement  process.  All  the  above  steps  culminate  in 
the  installation  and  operation  of  the  final  measure  set. 

t< 

In  closing.  Dr.  Muckier  reported  that,  with  respedt  to 
measurement,  there  is  the  implicit  assumption  that  if  more  data  is 
collected,  all  problems  will  be  solved.  This  concept  of  "if  it 
moves,  measure  it"  may  be  self-defeating:  the  issue  may  be 
confounded  rather  than  clarified.  Dr.  Muckier  also  suggested  that 
this  shotgun  approach  is  not  a  good  practice  and  he  suggested  that 
there  be  a  justification  for  all  data  points  collected. 

V' 
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[For  additional  information  on  performance  measurement 
development.  Dr.  Muckier  recommended  D.  Meister's  Human 
Performance  of  Systems,  in  preparation,  1986.] 

COMMENTS,  QUESTIONS  AND  ANSWERS  FOLLOWING  THE  TUTORIAL 

C.  I'm  curious  about  the  combined  overall  measures  that  Dr. 
McBride  alluded  to.  I  wonder  what  hope  we  have  of  getting  away 
from  time  and  accuracy,  frequency,  etc.  What  hope  do  we  have  for 
some  derived,  combined  numbers? 

McBride:  Obviously,  combined  measures  are  more  informative  than 

simple  measures.  Selection  of  a  measurement,  in  general,  is 
results  driven  --  a  top  down  approach.  The  idea  is  that  it  is  the 
original  question  which  drives  the  selection  of  performance 
criteria.  If  a  combined  measure  seems  to  be  more  relevant,  select 
the  combined  measure. 

Muckier:  One  of  the  most  exciting  theoretical  exercises  for  me 

has  been  developing  new  ideas  of  composite  measures;  and  in  the 
situations  I've  been  in,  there  is  a  large  amount  of  basic  data  and 
parameters  for  so  doing.  To  date,  I've  been  unsuccessful  in 
proposing  a  composite  measure  but  I  intend  to  keep  trying. 

Q.  Dr.  Rokicki  mentioned  going  from  optimal  human  performance  to 
"effective."  Could  he  explain  a  bit  more  of  what  he  means  by 
effective  and  how  effectiveness  is  measured? 

Rokicki:  Basically,  that  approach  arose  from  a  briefing  on  how  we 

are  operating  and  the  methodologies  that  we  are  using.  A  pilot 
doesn't  have  to  fly  an  airplane  at  total  peak  performance  all  the 
time:  he  has  to  be  capable  of  carrying  out  the  mission  but  he 

doesn't  have  to  perform  in  an  optimal  manner. 

However,  I  can't  really  answer  the  question  as  to  how 
effectiveness  is  measured  —  basically,  "effective"  means  the 
ability  to  complete  the  mission. 

C.  On  the  matter  of  error  measurement,  I  would  like  to  mention 
that  in  the  last  couple  of  years  I  have  encountered  two  situations 
where  it  was  not  at  all  clear  what  was  an  error  and  what  was  not. 
These  were  in  connection  with  an  anti-aircraft  missile  system  and 
an  anti-aircraft  gun  system.  Both  of  these  were  in  target-rich, 
hot  and  heavy  environments,  with  a  number  of  alternatives  that  the 
crews  could  pursue  at  given  points  in  the  engagement.  In  some 
cases  it  was  not  at  all  clear  that  one  alternative  was  right  or 
wrong,  or  even  that  one  was  better  than  the  other,  so  it's  not  a 
simple  problem. 

Q.  A  few  years  back,  the  Test  and  Evaluation  SubTAG  discussed 
the  difference  between  task  analysis  and  critical-task  lists. 
MANPRINT  seems  to  stress  the  need  for  critical-task  lists  and  my 
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question  to  Dr.  Geddie  is  do  we  have  to  go  back  to  a  fullblown 
task  analysis  or,  for  the  purposes  he  mentioned  in  his  briefing, 
would  a  critical-task  list  of  high  drivers  suffice? 

Geddie:  Probably  for  testing  and  evaluation,  a  list  of  critical 

tasks  --  not  a  fullblown  task  analysis  —  would  suffice.  I  would 
hope,  however,  that  the  prioritized  list  of  tasks  also  would  imply 
some  kind  of  performance  criteria  against  which  we  can  measure. 
Preferably,  we'd  be  able  to  measure  both  performance  time  and 
error  rate  and  have  some  criterion  to  compare  them  against.  If 
the  question  was  "is  a  fullblown  task  analysis  required?",  the 
answer  is  not  necessarily. 

C.  Would  Dr.  Geddie  enlarge  upon  his  comment  that  hardware 
failure  is  not  charged  to  system  design. 

Geddie:  That  comment  comes  from  my  experience  in  developmental 

testing.  In  the  reliability  community,  at  least  within  the  Army, 
there  has  been  a  10-12  year  battle  to  try  to  persuade  the  people 
who  calculate  system  reliability  and  predict  what  it  will  be  when 
a  system  is  fielded  not  to  regard  every  human  error  as  something 
that  is  not  chargable  against  hardware  design.  In  AMSA  we 
provided  two  calculations  of  systems  reliabillity .  One  of  which 
was  clearly  labeled  as  hardware  reliability.  The  other 
accommodated  human  error  as  part  of  the  number.  My  original 
comment  concerned  the  practice  of  doing  straight  hardware  testing 
which  disallows  any  kind  of  human  error  induced  failure. 

C.  A  specific  example  might  be  in  testing  a  tank  in  a  very  fluid 
battlefield.  Eight  targets  go  by  but  because  there  is  confusion 
over  the  status  of  four,  they  are  not  engaged.  The  last  four  are 
successfully  engaged.  In  this  example,  the  weapon  system  would  be 
scored  100%  effective  even  though  we  might  be  very  uncomfortable 
with  the  fact  that  half  the  time  the  target  went  by.  But  because 
the  system  wasn't  engaged,  the  first  four  were  not  charged  to  the 
system. 

Miles:  I  have  listened  to  some  very  articulate  arguments  from 

contractors  who  say  that  just  because  your  operator  tripped  over 
his  bootlace  and  therefore  didn't  engage  the  system,  you  can't 
charge  that  against  the  hardware.  The  hardware  was  there  and  was 
ready.  This  approach  misses  the  subtlety  of  the  fact  that 
sometimes  the  designs  of  the  hardware  induce  the  operator  to  make 
the  errors  that  cause  the  system  not  to  work. 

Muckier:  It  has  been  interesting  over  the  years  to  see , the  waxing 

and  waning  of  the  two  extreme  points  of  view  —  in  one,,  there  is 
no  such  thing  as  a  human  error.  The  other  side  says  that  if  a 
human  makes  an  error,  it  is  entirely  the  fault  of  the  equipment. 

o 

C.  I've  enjoyed  all  the  presentations  but  I  think  someone  should 
mention  the  fact  that  all  tests  and  all  evaluations  are  only  as 
good  as  the  initial  test  plan.  Also,  the  people  who  are 
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involved  in  the  testing  need  to  be  disciplined  and  very  much  aware 
of  the  importance  of  what  they  are  doing  and  not  interfere  with 
the  very  processes  they  are  measuring. 

Muckier:  Under  the  rules  of  the  game,  the  whole  process  starts 

with  the  requirements  document.  However,  in  nearly  one  system  out 
of  five,  there  is  no  requirements  document  despite  the  fact  that 
it  is  mandatory  or,  it  is  a  requirements  document  that  is  useless 
to  you.  We  often  have  to  make  it  up  as  we  go  along,  so  that  there 
will  be  a  requirements  document.  Also,  we  try  to  train  our  test 
people  as  rigorously  as  possible,  simulating  the  situation  before 
we  do  the  actual  testing.  After  12  hours  in  the  mud,  though,  it 
can  be  hard  to  be  disciplined. 

5.2  Implementation  Efforts  for  MANPRINT  —  Mr.  Judah 
Katznelson,  US  Army  Research  Institute  for  the  Behavioral  and 
Social  Sciences,  Alexandria,  Virginia 

See  Attachment  V  for  the  report  and  visual  materials 
submitted  by  Mr.  Katznelson. 

5 . 3  State-of-the-Art  in  AI/ES,  Decision  Aiding  and 
Augmentation  for  Advanced  Military  Systems  —  Dr.  Thomas  M.  Cook, 
Martin  Marietta  Aerospace  Company,  Denver,  Colorado 

View  Graphs  —  Attachment  W 

According  to  Dr.  Cook,  his  current  definition  of  artificial 
intelligence  is  the  attempt  to  produce  a  computer  system(s) 
(hardware,  software,  displays,  peripherals,  I/O  devices,  etc.) 
that  can  perform  complex  tasks  that  normally  require  human 
intelligence.  AI  seeks  to  enhance  the  performance  of  humans  in 
difficult  situations.  Related  areas  under  AI  include: 

expert  systems, 

natural  language  understanding, 

-  image  understanding  and  pattern  recognition, 
intelligent  robotics, 
advanced  computer  architectures,  and 
intelligent  decision  aids. 

Research  and  applications  in  the  expert  systems  area  are 
primarily  involved  with  the  development  of  programs  and  systems  to 
assist  the  human  and,  in  many  cases,  raise  human  performance  to 
the  level  of  a  true  expert.  Current  application  areas  in  which  a 
substantial  amount  of  money  is  being  invested  by  DOD  are  the  DARPA 
Strategic  Computing  Program,  the  Battle  Management  Program,  and 
the  Space  Station  Program. 

Relating  his  briefing  to  some  of  the  issues  discussed  during 
the  performance  measurement  tutorial,  Dr.  Cook  noted  that  the 
Autonomous  Land  Vehicle  (ALV)  program  provides  well  defined 
performance  measures.  Initially,  the  ALV  had  to  demonstrate  its 
ability  to  maintain  road  contact.  In  the  second  phase,  the  ALV 
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needed  to  demonstrate  that  it  could  maneuver  on  a  relatively 
straight  road  at  five  kilometers  per  hour;  and  within  the  next  few 
months,  ten  kilometers  per  hour  will  be  performance  parameter 
measured.  In  two  to  three  years,  it  is  expected  that  the  ALV  will 
be  able  to  demonstrate  its  ability  to  maneuver  across  rough 
terrain. 

Battle  Management  is  a  program  being  approached  by  all  the 
services.  The  three  major  areas  being  emphasized  are: 

1)  command,  control,  communications  and  intelligence  work; 

2)  staff  planning/replanning;  and 

3)  Pilot's  Associate  in  which  state-of-the-art  applications 
are  provided  to  the  pilot  to  assist  him  in  difficult  situations. 

Dr.  Cook  reported  that  there  are  a  number  of  artificial 
intelligence  applications  across  the  services  and  within  the 
Department  of  Defense  and  NASA.  These  applications  include: 

.  human-equipment  interface  aids  such  as  smart  displays, 
voice  input/output,  voice  recognition; 

.  planning  and  monitoring  aids  to  help  the  average  user 
achieve  a  level  of  expertise; 

.  expert  advisor:  providing  the  processes  and  information 
that  the  true  expert  uses  to  a  wide  variety  of  users; 

.  data  access  and  assimilation  (intelligent  database 
management ) ; 


.  support  systems  such  as  those  being  utilizd  in  all 

phases  of  the  launch/mission  control/recovery  process  of  space 
vehicles ; 


.  situation  assessment:  taking  large  amounts  of 

information  and  integrating  it  into  a  meaningful  analysis; 

.  system  controllers; 

.  information  collection;  and 

•  training  and  instruction,  using  intelligent  computer- 
aided  instruction. 

Dr.  Cook  reported  that  one  of  NASA's  major  thrusts  for  the 
development  of  AI  and  expert  systems  is  the  Space  Station  Program. 
According  to  Dr.  Cook,  Congress  has  mandated  that  ten  percent  of 
the  $8  Billion  Space  Station  budget  (design  and  building  phases) 
be  dedicated  to  advanced  automation  and  he  anticipates  that  at 
least  30  %  of  the  advanced  automation  aboard  the  Space  Station 
will  be  in  the  area  of  intelligent  systems.  At  this  point  in 
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time,  the  fault  management  area  is  receiving  the  most  emphasis. 
This  deals  with  detecting,  isolating  and  coping  with  a  particular 
fault  condition;  and  Dr.  Cook  reported  that  there  are  currently  20 
expert  systems  being  built  to  address  the  fault  management 
problem.  Other  AI  and  related  technologies  applications  within 
NASA  include:  the  satellite  monitoring  and  control  area, 
intelligent  robotics  and  the  development  of  intelligent  autonomous 
spacecraft. 

Dr.  Cook  discussed  some  of  the  problems  relating  to  available 
resources  for  AI  type  programs,  noting  that  there  is  a  significant 
shortage  of  trained  personnel  capable  of  building  AI  systems.  The 
majority  of  the  individuals  doing  AI  type  work  are  being  trained 
at  major  universities  such  as  M.I.T.,  Stanford,  Yale;  there  are  a 
limited  number  of  universities,  according  to  Dr.  Cook,  that  have 
good  programs  in  this  area.  Attempts  at  having  the  service  and 
NASA  train  their  own  AI  personnel,  while  successful,  have  created 
new  problems.  As  a  result  of  this  training,  the  experts  become 
quite  valuable  to  the  marketplace  and  NASA/government  agencies  can 
not  afford  to  retain  these  people.  In  addition,  a  number  of 
academics  are  going  into  industry,  creating  a  drain  on  the  basic 
research  community. 

One  of  the  other  difficulties  is  that  of  finding  the  "true 
expert"  and  capturing  his  knowledge  and  experience.  Dr.  Cook 
reported  that  there  have  been  cases  of  expert  systems  being  built 
where  the  individuals  making  the  inputs  were  more  "available"  than 
"expert."  Also,  since  there  are  relatively  few  true  experts,  they 
can  command  rather  large  fees:  getting  the  level  of  expertise 
necessary  has  become  a  very  expensive  situation. 

Dr.  Cook  indicated  that  the  hardware  necessary  to  build  very 
sophisticated  AI  applications  and  expert  systems  currently  exists. 
In  fact,  according  to  Dr.  Cook,  more  computing  power  is  available 
than  can  be  currently  utilized.  Dr.  Cook  stressed  that,  to 
implement  Al/expert  systems  programs,  the  government  and  NASA  will 
need  to  be  able  to  build  systems  that  are  small,  powerful, 
radiation-hardened,  and  which  can  accommodate  single-event  upsets. 

In  the  software  area  a  number  of  powerful  development  tools 
have  been  introduced  during  the  past  few  years;  and  Dr.  Cook 
expects  that  these  software  programs  will  enable  NASA  and  DOD  to 
take  individuals  who  are  not  fully  AI  trained  and  allow  them  to 
develop  some  of  the  application  areas. 

In  closing.  Dr.  Cook  outlined  a  technology  assessment  done 
for  the  Space  Station  Program,  noting  where  problems  and  solutions 
had  been  found. 

5 . 4  Crew  System  Ergonomics  Information  Analysis  Center 
(CSERIAC )  —  Mr.  Steve  Heckart,  Armstrong  Aerospace  Medical 
Research  Laboratory,  Wright-Patterson  Air  Force  Base,  Ohio 

View  Graphs  --  Attachment  X 
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Ergonomics  is  the  integrative  discipline  devoted  to 
understanding  and  quantifying  human  physiological  and  behavioral 
interaction  with  equipment  and  systems;  and  although  much 
pertinent  ergonomics  data  exist,  the  information  currently  is 
widely  dispersed  and  is  not  readily  accessible.  A  survey  of 
government,  industry  and  academic  crew  station  ergonomics  data 
users  indicated  that  a  central,  responsive  DOD  focal  point  in  crew 
station  ergonomics  is  needed.  Of  those  responding  to  the  survey, 
97%  use  crew  station  ergonomics  data  in  their  work  and  the 
products/services  they  need  most  are  : 

.  state-of-the-art  reviews  and  technology  assessments, 

.  rapid  response  to  technical  inquiries,  and 

.  current  awareness  bulletins. 

Eighty-eight  percent  of  the  respondents  supported  the 
establishment  of  CSERIAC  and  79%  indicated  a  willingness  to  pay 
for  CSERIAC 's  services. 

The  current  proposal  is  that  the  Air  Force  Armstrong  Aerospace 
Medical  Research  Laboratory  (AAMRL)  serve  as  the  host  of  this 
analysis  center  with  the  focus  being  on  crew  system  ergonomics  for 
military  aircraft  and  space  vehicles.  The  short  term  focus  of  the 
CSERIAC  is  on  airgoing  crew  systems.  However,  the  long-term  goal 
is  that  CSERIAC  would  provide  the  model  and  experience  base  to 
support  the  development  of  other  specialized  human  factors 
information  analysis  centers  hosted  by  DOD  agencies.  Thus,  in  the 
future,  it  is  possible  that  the  focus  of  CSERIAC  would  be  enlarged 
from  airgoing  crew  systems  to  include  the  needs  of  specific  users. 

Potential  users  include  government  laboratories,  evaluation 
agencies.  Systems  Commands,  the  Pentagon  and  the  logistics  and 
intelligence  communities.  Industry  users  might  include  prime  or 
subcontractors,  systems  and  operation  analysts,  and  the  controls, 
displays  and  software  communities. 

Subject  domains  of  CSERIAC  would  include  design  methodology, 
human  performance  characteristics,  information  display  interfaces, 
motor /control  systems  characteristics,  and  specialized-design 
issues . 

As  currently  envisioned,  the  functions/services  of  CSERIAC 
would  be  in  the  areas  of: 

.  rapid  expert  consultation  and  support, 

.  problem  solving  and  analysis, 

.  author  compendiums/handbooks  of  selected  ergonomics 

data , 

MIL  STANDARDS  and  SPECIFICATIONS,  and 
.  identification  of  high-value  technological 

gaps/voids . 
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CSERIAC  could  also  provide  a  recognized  liaison  with  a  variety  of 
user  communities  and  could  support  the  planning  and  conduct  of 
workshops,  symposia,  conferences  and  short  courses. 

5 . 5  Vision  in  Spatial  Disorientation  and  Loss  of  Aircraft 
Attitude  or  Control  Awareness  —  Colonel  Grant  McNaughton, 
Aeronautical  Systems  Division,  Wright-Patterson  Air  Force  Base, 
Ohio 


View  Graphs  —  Attachment  Y 

Colonel  McNaughton  noted  that  the  purpose  of  the  briefing  was 
to  provide  insight  into  how  orientation  and  attitude  information  is 
processed  and  to  point  out  some  of  the  design  features  that  impede 
valid  information  acquisition  and,  therefore,  impede  appropriate 
action.  Colonel  McNaughton  enlisted  the  support  of  the  TAG  for 
fomenting  pratical  improvements  in  present  aircraft  and  requested 
that  these  issues  receive  due  consideration  in  the  design  of  future 
aircraft.  He  also  indicated  that  the  aspects  covered  in  the 
briefing  should  be  of  use  to  those  who  are  involved  in  mishap 
prevention  and  mishap/accident  analysis. 

Historically,  spatial  disorientation  (SDO)  has  been  considered 
as  a  mismatch  between  vision  and  the  balance  organ.  However, 
current  information  indicates  that  the  mismatch  between  the  vision 
system  itself  and  its  two  modes  of  processing  visual  information 
(focal  and  ambient)  also  plays  a  major  role  in  SDO.  The  focal  mode 
is  an  object  recognition  system  that  is  exclusively  visual  and 
requires  good  lighting  and  optics  and  typically  involves  conscious 
attention.  The  ambient  mode  is  concerned  with  object  quality 
(quality  of  the  surrounds)  and  is  uncritical:  it  can  easily  be 
deceived.  This  mode  includes  vision,  balance,  proprioception  and 
hearing.  In  essence,  according  to  Colonel  McNaughton,  it  is  more 
of  an  ambient  orientation  system  in  which  vision  contributes  inputs 
along  with  those  from  the  other  senses.  The  ambient  system  carries 
most  of  the  workload  and  works  at  any  lighting  level.  Resolution 
is  totally  unimportant:  the  system  works  at  the  reflex  level. 

The  three  major  consequences  of  ambient  mode  reactions  involve 
distraction  potential,  vection  illusion,  and  false  horizons.  Since 
the  brain  contains  receptors  specifically  responsive  to  the 
components  of  motion  (velocity  and  direction),  any  motion  attracts 
the  eye.  These  responses  are  reflexive  and  extremely  difficult  to 
suppress.  Pilots  trained  to  detect  bogies  and  avoid  mid-air 
collision  are  especially  impacted  since  any  object  motion  can  also 
cause  head  motions  and  these  head  motions  can  further  add  to  the 
pilot's  disorientation. 

Whereas  a  small  amount  of  motion  on  the  periphery  can  be 
interpreted  as  object  motion,  more  of  the  peripherative  moves  can 
be  interpreted  as  "self"  motion.  This  vection  sensation  can  be 
true  or  illusory  and  is  the  principle  upon  which  full  visual 
simulators  are  based.  According  to  Colonel  McNaughton,  one  of  the 
design  features  that  potentiates  vection  illusions  is  a 
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bubble-shaped  canopy  in  which  the  position  of  the  pilot's  head  is 
such  that  his  ambient  mode  is  subjected  to  maximum  bombardment  — 
glare  reflections,  false  motions,  etc. 

Colonel  McNaughton  also  noted  that  findings  indicate  that  the 
visual  cortex  contains  receptors  specifically  responsive  to  lines 
and  edges.  Since  the  human  cannot  tolerate  a  sense  of 
disorientation  and  the  ambient  mode  is  uncritical,  it  is  likely  to 
accept  anything  with  the  quality  of  "horizoness . "  Since  it  appears 
as  if  the  larger  the  horizon,  the  more  commanding  it  is,  this 
impacts  the  way  aircraft  attitude  displays  should  be  designed. 
Colonel  McNaughton  reported  that  a  night  departure  across  a  lighted 
shoreline  is  particularly  lethal  because  the  balance  organ  cannot 
distinguish  between  acceleration  and  a  climb.  As  what  appears  to 
be  the  horizon  passes  beneath  the  pilot's  wingline,  he  becomes 
convinced  that  he  is  doing  a  loop  and  his  tendency  is  to  dump  the 
aircraft's  nose,  flying  back  into  the  water.  It  is  currently 
thought  that  surface  features  that  resemble  horizons  have  been 
responsible  for  a  number  of  mishaps. 

According  to  Colonel  McNaughton,  problems  with  current 
aircraft  include: 

.  wide-area  canopies; 

.  inadequate  attitude  references  (airframe  and 

instrument ) ; 

.  failure  to  provide  critical  control  parameters 

(airspeed,  altitude,  etc.)  in  a  digestible  format; 

.  lighting  problems; 

.  attention  traps  (caused  by  head  down  locations); 
and 

.  denial  of  important  non-focal  mode  sensory  cues 

(tactile  and  auditory). 

Colonel  McNaughton  discussed  mishaps  involving  particular 
aircraft,  noting  where  the  above  problem  areas  may  have 
contributed  to  the  mishaps. 

One  of  the  major  needs  of  the  pilot  to  help  him  recognize* 
combat  and  cope  with  the  problems  associated  with  spatial 
disorientation  and  false/no  horizon  is  that  of  the  enlargement  and 
position  of  the  attitude  indicator.  Colonel  McNaughton  stressed 
that  ADIs  need  to  be  prominent,  large  and  in  a  central  location. 
Ideally  they  would  be  large  enough  to  see  out  of  the  corner  of  the 
eye,  especially  when  flying  in  formation.  According  to  Colonel 
McNaughton,  all  current  ADIs  suffer  from  the  same  human  factors 
problem,  e.g.  the  roll  sensor  is  reversed;  and  when  under  great 
stress,  it  is  possible  for  even  experienced  pilots  to  become 
momentarily  confused. 

ft 
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Since  man  is  basically  a  pattern  recognizer,  the  more  one 
visually  organizes  his  information,  the  faster  he  is  able  to 
acquire  and  understand  the  information.  Colonel  McNaughton  noted 
that  a  pilot's  major  concern  when  looking  at  his  displays  is 
whether  the  parameters  represented  have  changed;  and,  if  so,  in 
which  direction  and  by  how  much.  The  problem  with  digital, 
alphanumeric,  and  symbolic  displays  is  that  they  require 
attentional  resources  (focal  mode)  to  read,  decode  and  integrate. 
These  displays  do  not  provide  inherent  trend  nor  limitation 
information.  On  the  other  hand,  properly  designed  analog  displays 
can  cue  both  the  focal  and  ambient  modes.  These  displays  provide 
the  information  more  quickly  and  can  also  provide  trend/limitation 
information.  However,  problems  occur  because  these  displays  are 
more  easily  misread. 

Some  of  the  problems  with  head  up  displays  (HUDs),  as 
currently  designed,  include  their  reliance  upon  the  focal  mode 
(digital  display),  the  amount  of  information  being  displayed 
(clutter)  and  the  methods/locations  in  which  the  information  is 
being  displayed  (confusion).  Head  up  displays,  in  Colonel 
McNaughton ' s  opinion,  are  not  basic  attitude  indicators,  since 
there  is  usually  a  limited  field  of  vision,  no  distinction  between 
the  sky  and  the  surface,  and  the  horizon  line  as  a  zero  degree 
pitch  line  is  no  more  commanding  than  any  other  pitch  scale.  In 
unusual  attitudes,  the  HUD  is  subject  to  misinterpretation;  and 
there  is  a  tendency  for  pilots  to  stare  at  the  HUD  without 
actually  processing  the  information. 

Colonel  McNaughton  recommended  that  the  following  aspects  be 
considered  in  improving/modifying  current  aircraft: 

HUD  (instrument  mode) 


.  utilize  a  vertex  pointer  on  the  flight  path  marker; 

.  radically  change  the  pitch  scale  pattern  between  positive 
and  negative  and  within  negative; 

.  consider  color  change(s)  as  redundant  cue(s); 

.  lengthen  all  pitch  scale  lines;  and 

.  consider  a  patterned  analog  format  for  parameters  such 
as  the  airspeed  indicator,  attitude,  vertical  velocity 
indicator,  angle  of  attack  and  gravity. 

ADI 

.  provide  dedicated  attitude  indicator; 

.  located  high  in  the  center  of  the  instrument  panel; 

.  consider  HUD  control  panel  area  for  projected  image; 
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.  colocate  ADI  and  Standby  Attitude  Indicator  for  rapid 
mismatch  recognition; 

.  improve  nonvisual  cues  to  airspeed  (radical  pattern 
changes  in  different  ranges); 

.  improve  spot  lighting; 

.  reduce  glare  and  reflections  (glare  shields 

extensions/batwing  shades); 

.  utilize  a  ground  proximity  warning  system;  and 

.  utilize  a  radar  altimeter. 

Colonel  McNaughton 1 s  suggestions  for  improvement  in  future 
aircraft  included: 

make  improvements  in  the  HUD  for  attitude,  as  noted 
above ; 

utilize  a  dedicated  ADI  on  the  HUD  control  panel; 

utilize  a  dedicated  ADI/SAI  below  in  HUD  control  panel 
(different  information  source); 

optimize  mixture  of  sensory  inputs;  and 

consider  mission  diversity  (basic  instrument  flying, 
night  formations,  etc.). 

In  closing.  Colonel  McNaughton  reiterated  that  man  is  a 
creature  of  habit:  special  care  needs  to  utilized  when  locating 
switches,  ejection  handles,  and  critical  instruments.  Designers 
also  need  to  give  serious  consideration  to  providing  the  pilot  with 
the  necessary  attitude,  airspeed,  altitude,  etc.  cues  in  an  easily 
recognizable  and  understandable  manner  so  that  his  focal  mode 
processing  is  not  tied  up;  thereby,  freeing  the  focal  mode  for 
those  tasks  specifically  requiring  the  focal  mode.  By  lessening 
the  focal  mode  workload,  situational  awareness  is  improved  and  the 
potential  for  spatial  disorientation  is  reduced. 

[See  also  4.2.1  and  5.6,  these  Minutes.] 

[It  was  not  possible  to  include  all  of  the  specific  details 
noted  by  Colonel  McNaughton  in  his  presentation.  Individuals 
desiring  a  copy  of  briefing  tapes,  should  contact  the  TAG 
Coordinator,  Louida  D.  Murray,  at  (303)  979-7441.] 

5.6  Head  Up  Displays  as  a  Primary  Flight  Instrument  --  Dr. 
Richard  J.  Schiffler,  Aeronautical  Systems  Division, 
Wright-Patterson  Air  Force  Base,  Ohio 
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Over  the  past  few  years,  there  has  been  considerable  concern 
as  to  the  role  head  up  displays  play  in  aircraft  accidents.  In 
June  1983,  Headquarters  Tactical  Air  Command  held  a  conference  to 
address  HUD  issues.  At  that  point  in  time,  TAC  policy  was  that  a 
HUD  was  not  considered  to  be  a  primary  flight  instrument.  Other 
concerns  discussed  at  the  conference  were  the  degradation  of 
instrument  flying  skills  and  methods  of  training  pilots  to  use  the 
HUD. 


In  June  1985  a  USAF-wide  conference  was  held  at  Langley  Air 
Force  Base  with  the  primary  purpose  being  that  of  the  development 
of  a  USAF  policy  on  HUDs  and  instrument  flying.  Presentations 
were  made  by  the  Air  Force  Inspection  and  Safety  Center,  the 
Federal  Aviation  Administration,  Aeronautical  Systems  Division/EN, 
Tactical  Air  Command,  the  Instrument  Flight  Center  and  a  number  of 
laboratories.  Although  the  conference  did  not  formuJLate  a 
definition  of  what  constitutes  a  primary  flight  instrument,  there 
were  a  series  of  recommended  policies  regarding  HUDs.  These 
recommendations,  now  under  review,  include: 

.  any  instrument  flight  system  (including  HUD  and  head 
down  displays)  should  include  control,  performance,  navigation  and 
failure  information;  and 

.  head  down  information  is  still  needed  as  backup 
information. 

Dr.  Schiffler  reported  that  33  F-16  pilots  at  Nolles  Air 
Force  Base  and  35  A-10  pilots  at  Myrtle  Beach  Air  Force  Base  were 
surveyed  in  1982-83  in  an  attempt  to  determine  eye  position  during 
different  mission  segments.  Interview  data  was  also  collected  on 
the  percentage  of  time  the  HUD  was  used  during  different  mission 
segments.  In  the  F-16,  low  use  of  the  HUD  was  reported  during  the 
take  off,  climb,  cruise  and  low  level  segments.  In  the  weapon 
delivery  segments,  the  HUD  was  used  as  a  primary  instrument.  This 
finding  is  consistent  with  expectations,  considering  the  type  of 
information  displayed  on  current  HUDs.  Surveyors  did  note  that  in 
the  approach  and  landing  mode  the  HUD  was  equally  used  across  the 
low  and  median  high. 

In  the  case  of  the  A-10  pilots,  results  were  similar:  only 
in  the  weapon  delivery  mode  was  there  high  HUD  use  as  a  primary 
instrument.  The  pattern  in  the  approach/landing  mode  was 
different  from  that  of  the  F-16  group.  Dr.  Schiffler  indicated 
that  these  differences  may  be  due  to  the  nature  of  the  A-10  HUD. 
This  older  HUD  has  a  narrower  field  of  view  than  does  the  F-16  and 
there  is  less  symbology  information  displayed. 

Conclusions  drawn  from  the  review  of  the  survey  data  were: 

.  overall,  the  use  of  the  HUD  is  normally  distributed; 

.  there  is  high  use  of  the  HUD  during  the  weapon  delivery 

task ; 
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.  there  is  low  use  of  the  HUD  during  takeoff,  climb, 
cruise,  and  low  level  flights; 

.  there  is  no  consistent  relationship  between  HUD  use  and 
pilot  experience; 

.  the  F-16  HUD  tends  to  pull  pilots  forward  in  the 

cockpit,  thus  making  more  information  available;  and 

.  most  pilots  prefer  a  20°  field  of  vision. 

Many  problems  with  the  HUD  need  to  be  resolved  if  it  is  to  be 
utilized  as  a  primary  flight  instrument.  The  HUD  does  not  provide 
adequate  attitude  information  and  the  compelling  quality  of  the 
HUD  may  interrupt  normal  crosschecking  processes.  Also,  vertigo 
may  be  induced  if  it  is  necessary  for  the  pilot  to  switch 
repeatedly  from  a  head  up  to  a  head  down  position  to  access  all 
the  pertinent  information.  The  HUD  has  a  narrow  field  of  view, 
limiting  the  peripheral  cues  necessary  to  determine  spatial 
relationships  and  the  location  of  enemy  aircraft.  In  addition, 
the  amount  of  information  displayed  may  made  the  HUD  difficult  to 
read  during  certain  regimes  and  current  failure  modes  are 
inadequate. 

Dr.  Schiffler  noted  that  the  F-18  HUD  has  advantages  over  the 
F-16  HUD  in  that  the  F-18  HUD  has  a  horizon  line,  a  pointer 
indicates  position  in  space,  and  roll  information  is  included  on 
the  bottom  of  the  F-18  HUD.  The  Navy  has  a  major  effort  underway, 
studying  in  part,  the  F-18  HUD  as  a  primary  flight  instrument. 
According  to  Dr.  Schiffler,  the  F-16  System  Program  Office  has 
tasked  ASD  with  addressing  these  issues  as  they  relate  to 
modifying  the  present  F-16  HUD  to  make  it  a  primary  instrument. 

ASD  will  utilize  its  man-in-the-loop  simulation  crewstation  design 
facility,  testing  operational  pilots  with  differing  experience 
levels.  In  the  F-16  simulator,  HUD  information  is  generated  and 
superimposed  on  a  closed  circuit  television  system.  Pilots  fly 
mission  scenarios  and  the  investigators  collect  performance  and 
workload  data.  A  variety  of  studies  have  been  planned,  utilizing 
different  methods  of  presenting  roll  information,  altitude  and 
attitude  information,  etc. 

In  conclusion.  Dr.  Schiffler  reiterated  that  HUDs  have  been 
and  will  continue  to  be  used  in  a  primary  instrument  fashion. 

More  research  must  be  done  to  determine  what  information  the  pilot 
needs,  head  up  or  head  down,  to  recover  from  various  flight 
regimes . 

I 

5 . 7  Human  Factors  and  Organizational  Systems  Laboratory, 

Navy  Personnel  Research  and  Development  Center  —  Dr.  Robert  E. 
Blanchard,  Navy  Personnel  Research  and  Development  Center,  San 
Diego,  California 

Navy  Personnel  Research  and  Development  Center  is  the 
principal  Navy  research,  development,  testing  and  evaluation 
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center  for  manpower,  personnel,  education,  training,  and  human 
factors  and  provides  technical  support  to  the  Chief  of  Naval 
Operations  in  these  areas. 

The  Center's  basic  goal  is  to  perform  both  field  and 
laboratory  RDT&E  and  simulation  work  that  will  increase  the  Navy's 
readiness  and  effectiveness.  Current  and  anticipated  problems  in 
manpower  management,  personnel  administration,  organization 
management,  education,  training,  and  human  performance  are 
considered.  The  Center  is  organized  around  three  R&D 
laboratories:  (1)  Training;  (2)  Manpower;  and  (3)  Human  Factors 

and  Organizational  Systems. 

The  Human  Factors  and  Organizational  Systems  Laboratory  is 
concerned  with  developing  and  conducting  an  R&D  program  to  advance 
the  behavioral  technologies  supporting  an  improved  understanding 
of  man's  interaction  with  other  individuals  and  complex  hardware 
systems.  The  program  addresses  social,  technical,  and  physical 
environmental  factors  for  enhancing  performance  and  improving  the 
quality  of  working  life. 

Since  the  Navy  is  increasingly  dependent  upon  complex  combat 
and  support  systems,  the  Human  Factors  Department  within  this 
laboratory  has  responsibility  for  developing  and  conducting  R&D  to 
extend  the  fundamental  knowledge  of  human  processes  underlying 
advanced  concepts  and  models  of  human-machine  functions. 

Resulting  principles,  techniques,  and  data  are  organized  and 
applied  with  an  integrated  systems  approach  that  seeks  to  optimize 
the  design,  development,  operation,  and  maintenance  of  Navy  human- 
machine  systems.  Major  program  activities  include: 

* 

1.  Providing  human  factors  support  to  the  development  of 
command  and  control  (C2)  systems. 

2.  Expanding  the  application  of  human  factors  principles  to 
the  design  and  development  of  new  systems. 

3.  Providing  RDT&E  to  augment  and  stimulate  human  factors 
efforts  in  the  design  and  development  of  new  systems. 

4.  Improving  the  effectiveness  of  individuals  and  teams  by 
application  of  principles  of  human  performance. 

5.  Ensuring  the  application  of  human  factors  principles  and 
data  to  Navy  systems  throughout  their  life  cycle. 

6.  Applying  mathematical  models  to  enhance  decision-making 
capabilities  of  advanced  combat  systems. 

To  achieve  maximum  readiness  within  the  limits  of  available 
resources,  the  Navy  must  also  develop  the  full  potential  of  its 
human  resources  and  apply  that  potential  toward  maximum 
productivity.  The  Organizational  Management  Systems  Department 
has  this  responsibility.  This  program  develops  and  conducts  R&D 
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addressing  individual  and  organizational  processes  as  they  relate 
to  organizational  functioning  and  the  measurement  of  attitudes, 
perceptions,  expectations,  and  performance  of  military  and 
civilian  Navy  personnel  in  organizational  contexts.  The  human 
processes  are  assessed  relative  to  motivation  and  productivity  and 
new  techniques  and  strategies  are  developed  and  applied  for 
improving  overall  organizational  effectiveness.  Major  program 
activities  include: 

1.  Identifying  and  developing  appropriate  measures  and 
reporting  systems  of  both  individual  and  organizational 
performance. 

2.  Investigating  the  factors  and  policies  that  facilitate 
or  inhibit  the  Navy's  organizational  functioning. 

3 .  Examining  the  impact  of  new  technology  on  workers  and 
organizations  and  determining  what  is  required  to 
realize  expected  benefits  from  the  implementation  of  new 
technology. 

4.  Developing  principles  and  techniques  for  evaluating  and 
enhancing  command  effectiveness. 


6.  CHAIR'S  SUMMARY 

Mr.  Crites  expressed  his  appreciation  to  the  Navy  Personnel 
Research  and  Development  Center  for  hosting  the  TAG  and  noted  that 
the  efforts  of  CDR  John  Kohler,  Dr.  Robert  Blanchard  and  LT  Edward 
Trautman  had  contributed  to  the  productivity  and  cordiality  of  the 
meeting.  Mr.  Crites  noted  that  time  constraints  have  forced  some 
of  the  SubTAGs  to  schedule  additional,  off-line,  meetings  and  that 
it  may  be  necessary  for  the  subgroups  to  schedule  Monday  and/or 
Friday  meetings  during  TAG  week. 

Mr.  Crites  reported  that  the  TAG  would  be  responsive  to 
OUSDR&E ' s  request  for  information  concerning  cockpit  automation 
technology.  In  closing,  he  thanked  the  guest  speakers,  attendees 
and  Ms.  Louida  D.  Murray  for  their  inputs  to  the  meeting. 
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1830  -  2000 

Controls  and  Displays 

Mr.  Grossman 

Surf 

1830  -  2000 

Professional  Educat i on 

Mr.  Jones 

Crystal 

1830  -  2000 

LOGSTAG 

Mr.  Mahar 

Teak 

2010  -  2100 

Operat i ng  Board 

Mr.  Crites 
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^Registration  materials  (badges,  receipts)  will  be  available  at 

the  TS/I 

meet i ng 

in  the  Crystal  Room.  Coffee 

and  danish  wi 1 1  be  served 

dur i ng 

tne  meet 

mg.  Pre-registration  and 

reservations  are  REQUIRED, 

■V  ‘.EETING  IN 

CCNJINCTICN  WITH  TaG  15: 

Sus  Ops  --  LTC  6.  Krueger,  (301)  427-5521 /AW  291-5521. 


Aviation  Screening  —  Dr.  M.  Sanders,  (205)  255-6862/ 
AW  558-6862 


go ,  C* 1 i t or n i * 
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Wednesday.  Nou ember  o  6a*  Uiew  Room 


0830  - 
0900  - 

0505  - 


1030  - 

10  50  - 
1:00  - 

1130  - 
:  1 4Q  - 

1150  - 


1200  - 


1330  - 
1430  - 
1440  - 
1450  - 
1500  - 
1515  - 


1545  - 
1615  - 
1890  - 


0900  COFFEE,  MUFFIN',  Registration 

0905  Navy  Personnel  Research  and 
Development  Center  Ue leone 
1020  Tutorial:  Selecting  Performance  Measures  i 

HFE  Testing  and  Research 

Moderator:  Dr.  John  Miles,  Jr.  (USA) 

Panelists:  LCDR  Dennis  McBride  <USN) 

Dr.  Frederick  Muckier  (APA/21) 
Dr.  James  Geddie  (USA) 

Dr.  Stephen  RoKicKi  (USAF) 

1050  COFFEE 

1100  Administrative 

1130  Implementation  Efforts  for  MANPR1MT 

1140  Workload  Report 

1150  Manned  System  Modeling  Report 

1200  Professional  Education  Report 

1320  LUNCHEON  BREAK 

1430  TBD 

1440  Test  and  Evaluation  Report 

1450  HFSSC  Report 

1500  LOGSTAG  Report 

1515  COFFEE 

1545  State  of  the  Art  in  AI/ES,  Decision 

Aiding  and  Augmentation  for  Advanced 
Mi  1 i tary  Systems 

1615  Crew  System  Ergononics  Information 

Analysis  Center  (CSER1AC) 

1625  HFE-GSED  Report 

1°30  Cocktail  Party  (reservations  required) 


CDR  John  Kohler 


Mr.  Judah  Kat:neison 
Mr.  James  Hartzel 1 
Ms.  Heidi  Fiedler 
Mr.  Todd  Jones 


TBD 

MR.  Keith  Karri 
Mr.  Gerald  Chaikin 
Mr.  Dale  Mahar 

Dr.  Thcmas  Cook 


Mr.  Steve  Heckart 
Mr.  Steve  Heckart 
Teak  Room 


A-2 


TN*jr«oi/,  November  7  Bay  View  Room 

oeoo  - 

0830 

COFFEE.  MUFFINS 

0630  - 

0*00 

Vision  in  Spatial  Disorientation  and  Loss 

Col .  Grant  McNaughton 

of  aircraft  Attitude  or  Control  Awarness 

0*00  - 

0910 

Controls  and  Displays  Report 

Mr.  Jeffrey  Grossman 

0*10  - 

0920 

Sustained/Continuous  Operations  Report 

LTC.  Gerald  Krueger 

0920  - 

0950 

Analysis  of  Aircraft  Avionics  Integration 

Mr.  Cyrus  Cri tes 

0°50  - 

1000 

Voice-Interactive  Systems  Report 

Mr.  Clay  Coler 

1000  - 

1015 

COFFEE 

10!?  - 

1045 

Overview  of  Navy  R&D  Efforts  at  NPRDC 

Dr.  Robert  Blanchard 

1045  - 

1100 

Technical  Society/Industry  Report 

Dr.  Frederick  Muckier 

1100  - 

1110 

User-Computer  Interaction  Report 

Mr.  Larry  Peterson 

1110  - 

1120 

Army  Report 

Mr.  Clarence  Fry 

1120  * 

1 1 30 

Navy  Report 

LCDR  Thomas  Mi  tchel  1 

1130  - 

1140 

Air  Force  Report 

Dr.  Richard  Schiffler 

1140  - 

1150 

NASA  Report 

Mr.  John  Barren 

1150  - 

1200 

Chair's  Summary 

Mr.  Cyrus  Crites 

1200  - 

1330 

LUNCHEON  BREAK 

1330  - 

1530 

NPRDC  Tour 

1345  - 

1500 

Executive  Comni ttee  Meeting 
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ATTACHMENT  B 


TAG  OPERATING  STRUCTURE 


OPERATING  STRUCTURE 


GOALS 


DEPARTMENT  OF  DEFENSE 

HUMAN  FACTORS  ENGINEERING  TECHNICAL  ADVISORY  GROUP 


Provide  a  mechanism  for  exchange  of  technical  information  in  the  develop¬ 
ment  and  application  of  human  factors  engineering. 

Enhance  working-level  coordination  among  Government  agencies  involved  in 
HFE  technology  research,  development,  and  application. 

Identify  human  factors  engineering  technical  issues  and  technology  gaps. 

Encourage  and  sponsor  in-depth  technical  interaction,  including  subgroups 
as  required  in  selected  topical  areas. 

Assist  as  reauired  in  the  preparation  and  coordination  of  triservice 
documents  such  as  Technology  Coordinating  Papers  and  Topical  Reviews. 

SCOPE 


Because  of  the  diversity  of  subject  matter  covered  by  the  HFE  discipline, 
the  scope  of  technical  areas  addressed  by  the  Technical  Advisory  Group  (TAG) 
is  necessarily  broad.  In  general,  HFE,  as  defined  for  purposes  of  TAG 
operation,  deals  with  concepts,  data,  methodologies,  and  procedures  which  are 
relevant  to  the  development,  operation,  and  maintenance  of  hardware  and 
software  systems.  Subject  matter  subsumes  all  technologies  aimed  at 
understanding  and  defining  the  capabilities  of  human  operators  and  maintainers 
and  insuring  the  integration  of  the  human  component  into  the  total  system  to 
enhance  systems  effectiveness.  Technologies  directed  toward  improved  manpower 
utilization  through  selection,  classification,  and  training  are  included  as 
appropri ate. 


TOPICAL  AREAS 

The  TAG  will  address  research  and  technologies  designed  to  impact 
man-machine  system  development  and  operation  throughout  the  complete  system 
life-cycle.  The  general  topics  of  concern  to  the  TAG  include,  but  are  not 
limited  to: 


a.  Procedures  for  use  by  HFE  specialists,  systems  analysts,  and  design 
engineers  involved  in  the  provision  of  HFE  support  during  system 
development  or  modification. 
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b.  Methodologies  oriented  toward  the  identification  and  solution  of 
operator/maintainer  problems  related  to  equipment  design,  operation, 
and  cost/effectiveness. 

c.  Mechanisms  for  application  of  developed  HFE  technologies,  including 
formal  and  informal  approaches  to  validation  and  implementation,  and 
the  determination  of  time  windows  for  application. 

GROUP  COMPOSITION 

The  TAG  will  consist  of  technical  representatives  from  Government  agencies 
with  research  and  development  responsibility  in  the  topical  areas  specified 
above.  Additional  representatives  from  activities  with  allied  interests  may 
affiliate  with  the  TAG  as  appropriate.  Attendance  at  specific  meetings  may  be 
augmented  by  technical  experts  in  special  topical  areas. 

OPERATING  BOARD 


The  TAG  Operating  Board  is  responsible  for  the  conduct  of  TAG  business 
and  the  implementation  of  TAG  policies.  The  Board  consists  of  an  Executive 
Committee,  the  chairpersons  of  all  SubTAGs  and  Committees,  and  liaison 
representatives  from  selected  Government  agencies.  Operating  Board  meetings 
are  called  at  the  discretion  of  the  TAG  Chair. 

The  Executive  Committee  will  be  responsible  for  providing  required 
continuity  and  acting  for  the  full  TAG  between  regular  meetings.  Regular 
members  of  the  Executive  Committee  will  be: 

o  Current  Chair  o  Army  Representative 

o  Immediate  Past  Chair  o  Navy  Representative 

o  Chair  Select  o  Air  Force  Representative 

o  NASA  Representative 


CONDUCT  OF  BUSINESS 


Meetings  of  the  TAG  will  be  held  semi-annually,  in  the  Spring  and  the 
Fall.  Chairing  of  the  group  will  rotate  annually  among  the  Army,  Navy,  and 
the  Air  Force.  The  Chair  Select  will  be  chosen  by  a  caucus  of  the 'service, 
whose  turn  it  is  to  chair  the  DOD  HFE  TAG.  Advice  and  counsel  will  be 
provided  by  the  Operating  Board.  The  Service  Representatives  will  be  selected 
by  service  caucus  at  the  Spring  meetings  in  even-numbered  calendar  years. 
Advice  and  counsel  will  be  provided  by  the  Operating  Board.  Minutes  of  each 
meeting  will  be  compiled  by  the  Chair.  Minutes  will  be  distributed  to  all 
participants,  to  appropriate  OSD  offices,  and  to  other  agreed-upon  agencies. 
Minutes  shall  serve  as  the  principal  mechanism  for  the  reporting  of  group 
activity.  A  file  of  minutes  and  relevant  correspondence  shall  be  maintained 
by  each  Chair.  This  file  shall  be  passed  to  the  succeeding  Chair  together 
with  any  additions  to  the  file. 
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TAG  SUBGROUPS 


The  DOD  HFE  TAG  sactions  two  categories  of  subgroups:  SubTAGs  and 
Committees.  Such  groups  will  be  sponsored  by  the  TAG  as  appropriate  to 
respond  to  needs  for  more  detailed  interchange  and  coordination  in  specific 
technical  areas.  SubTAGs  will  address  problems  of  a  general  or  continuing 
nature  within  a  specific  field  of  technology  and  are  to  develop  their  own 
working  charters  and  operating  procedures.  SubTAGs  may  be  disestablished  upon 
recommendation  of  the  Executive  Committee.  Committees  will  serve  at  the 
pleasure  of  the  Operating  Board  and  will  address  specifically  defined  tasks  or 
problems.  These  committees  will  be  disestablished  on  completion  of  those 
tasks  or  upon  recommendation  of  the  Executive  Committee.  Reports  from  each 
subgroup  will  be  published  separately  and  included  as  a  regular  item  of 
business  on  each  TAG  meeting  agenda.  Current  subgroups  are  identified  in 
Appendix  A. 
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ATTACHMENT  C 


TAG  POLICIES 


TAG  POLICIES 


1.  Membership  (General  membership  policies  are  outlined  in  the  Operating 
Structure,  under  "Group  Composition.") 

1.1  Individuals  who  are  not  affiliated  with  Government  agencies  but  are 
associated  with  technical  societies  or  industrial  associations  with  a  stated 
interest  in  human  factors  engineering  are  to  submit  a  letter  on  the 
organization's  letterhead,  confirming  their  status  as  the  organization's 
representative,  to  the  current  chairperson  of  the  Technical  Society/ Industry 
Committee. 


2.  Meeting  Sites  (Sites  are  selected  by  the  Executive  Committee  with  a  view 
toward  a  balance  in  geographic  location,  service  hosting  the  meeting,  and 
meeting  facilities. ) 

2.1  Tag  members  are  encouraged  to  recommend  potential  meeting  sites. 

2.2  Organizations  who  wish  to  host  the  TAG  should  contact  their  Service 
Representative  or  the  current  TAG  Chair. 


3.  Agenda  (The  agenda  is  determined  approximately  two  months  before  the 
scheduled  meeting.  The  Executive  Committee  selects  the  topics  from  those 
recommended  by  the  Service  Representatives  and  the  current  TAG  Chair.) 

3.1  TAG  members  are  encouraged  to  suggest  potential  agenda  topics  or 
topics  suitable  for  tutorial  sessions  to  their  Service  Representative  or  to 
the  current  TAG  Chair. 


4.  Registration  (Registration  fees  and  the  date  of  the  close  of  registration 
are  announced  in  an  information  letter  sent  approximately  one  month  before  the 
scheduled  meeting.) 

4.1  All  attendees  are  expected  to  pre-register  and  prepay  by  mail. 

4.2  Individuals  receiving  late  travel  approvals  may  pre-register  by 
phone  by  contacting  the  TAG  Coordinator  identified  in  the  TAG 
invitation  letter.  All  payments  made  at  the  meeting  site  are  to  be 
in  cash. 


5.  Minutes  (The  Minutes  of  each  meeting  serve  as’the  principal  mechanism  for 
the  reporting  of  TAG  activities.  The  Minutes  are  published  as  a  draft 
document  and  distributed  to  attendees  and  other  selected  agencies 
approximately  three  months  after  the  meeting.) 
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5.1  Individuals  or  agencies  desiring  to  be  included  on  the  distribution 
list  for  a  specific  meeting  should  contact  the  TAG  Coordinator. 

5.2  Amendments  to  the  Minutes  are  to  be  made  to  the, TAG  Chair  in  writing 
prior  to  the  succeeding  meeting. 

5.3  Presentors  are  expected  to  submit  a  copy  of  their  presentations  and 
hardcopies  of  their  visual  materials  to  the  TAG  Coordinator  for  inclusion  in 
the  Minutes. 


6.  Subgroups  (See  the  Operating  Structure,  section  entitled  "TAG  Subgroups," 
for  specific  information  regarding  the  purposes  and  operating  procedures  of 
SubTAGs  and  Committees.) 

6.1  All  subgroups  are  strongly  encouraged  to  meet  in  conjunction  with 
the  TAG  at  least  once  each  calendar  year. 

6.2  All  subgroups  meeting  in  conjunction  with  the  TAG  are  required  to 
provide  a  chairperson  for  the  specific  subgroup  meeting. 

6.3  All  subgroup  chairpersons  are  encouraged  to  submit  a  brief  report  of 
each  meeting  to  be  included  in  the  set  of  TAG  Minutes  covering  the  subgroup 
meeting  timeframe. 

6.4  All  subgroups  are  required  to  provide  the  TAG  Coordinator  with  an 
up-to-date  list  of  their  membership,  for  use  in  the  distribution  of  TAG 
announcements. 

6.5  All  SubTAGs  are  required  to  submit  to  the  Executive  Committee  a 
Charter  including,  but  not  limited  to,  statements  regarding: 

o  objectives  o  membership  policies  o  meeting  schedule 

o  scope  o  chairperson 

6.6  Committees  are  required  to  submit  to  the  Executive  Committee  a 
document  including,  but  not  limited  to,  brief  statements  regarding: 

o  objectives 
o  membership  policies 
o  chairperson 


7 .  Subgroup  Establishment 

7.1  Groups  interested  in  addressing  technical  areas  not  covered  by 
existing  subgroups  may  request  the  TAG  Chair  to  provide  subgroup 
agenda  meeting  time. 

7.2  Formal  subgroups  may  be  established  by  recommendation  of  the 
Executive  Committee. 
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8.  Chair/Representative  Selection  (General  selection  procedures  are  outlined 
in  the  Operating  Structure  under  "Conduct  of  Business.") 

8.1  A  Service  caucus  may  be  called  by  the  TAG  Chair  or  the  current 
Service  Representative. 

8.2  Methods  of  determining  the  Chair  Select  and  Service  Representatives 
are  Service  peculiar. 

8.3  Unexpired  terms  of  office  will  be  filled  by  appointment  by  the 
Executive  Committee,  until  a  caucus  of  the  Service  can  be  called  at 
the  next  regularly  scheduled  meeting. 


9.  Fundi nq  The  funding  required  for  the  organization,  conduct,  and 
documentation  of  all  TAG  meetings  shall  be  done  jointly  be  the  three 
services.  The  specific  mechanisms  to  obtain  and  allocate  funding  from 
services  shall  be  arranged  by  the  Current  Chair,  Chair  Select  and  Immediate 
Past  Chair. 


10.  Pol  icy  Changes 

10.1  Additions  to  or  amendments  of  the  above  policies  may  be  recommended 
by  submitting  the  suggested  change(s)  in  writing  to  the  TAG  Chair. 

10.2  Policies  may  be  amended  by  a  majority  vote  of  those  Operating  Board 
members  in  attendance  at  the  Operating  Board  meeting  in  which  amendments  have 
been  proposed. 
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ATTACHMENT  D 


TAG  OPERATING  BOARD 


TAG  OPERATING  BOARD 
(1/15/86) 


* 


Executive  Committee 


Current  Chair  (Air  Force) 


Mr.  Cyrus  D.  Crites 
6520  TEST-ENAH 
Stop  239 

Edwards  AFB ,  CA  93523 
(805)  277-3334 
AV  350-3334 


Immediate  Past  Chair  (Navy) 
Mr.  Paul  M.  Linton 
Sikorsky  Aircraft 
MS-S326A 
N.  Main  Street 
Stratford,  CT  06601 
(203)  386-6315 

Navy  Seat 

Dr.  John  O' Hare 

Office  of  Naval  Research 

Code  442 

800  N.  Quincy  St. 
Arlington,  VA  22217-5000 
(202)  696-4502 

^  AV  226-4502 

Chair  Select  (Army) 

Dr.  Michael  H.  Strub 

P.O.  Box  6057 

ARI  Field  Unit 

Fort  Bliss,  TX  79916-0057 

(915)  568-4491 

AV  978-5297 


Army  Representative 
Mr.  Clarence  Fry 
Director,  USA  Lab  Command 
Human  Engineering  Lab;  SLCHE-AD 
(Fry) 

Aberdeen  Proving  Ground,  MD 
21005-5001 
(301)  278-5834 
AV  298-5834 

Navy  Representative 

LCDR  Thomas  M.  Mitchell 
Pacific  Missile  Test  Center 
Code  4025 

Point  Mugu,  CA  93042 
(805)  989-9424 
AV  351-9424 

Air  Force  Representative 
Dr.  Richard  J.  Schiffler 
ASD/ENECH 

Wright-Patter son  AFB,  OH  45433 
(513)  255-5597 
AV  785-5597 

NASA  Representative 

Dr.  Melvin  D.  Montemerlo 
Code  RC 

NASA  Headquarters 
Washington,  DC  20546 
(202)  453-2743 
FTS  453-2743 


Ex  officio  Members 
SubTAG  Chairs 


Controls  and  Displays  (Displays) 
Mr.  Jeffrey  D.  Grossman 
Commander 

Navy  Personnel  R&D  Center 
Code  71 

San  Diego,  CA  92152 
(619)  225-2081/2 
AV  933-2081 


Human  Factors  in  Aviation  Screening  and  Performance  Prediction 
(Aviator  Screening) 

Dr.  Michael  G.  Sanders 
USA  Aeromedical  Research  Division 
Ft.  Rucker,  AL  36362 
(205)  255-6862 

AV  558-6862  D-l 


Human  Factors  in  Logistics  (LOGSTAG) 

Mr.  Dale  Mahar 

Pacific  Missile  Test  Center 

Code  4025,  Bldg  7020 

Pt .  Mugu,  CA  93042 

(805)  989-9511 

AV  351-9511 

Human  Factors  Test  &  Evaluation  (T&E) 

Dr.  James  C.  Geddie 
USAHEL  Field  Office 
HQ  TCATA 

ATTN:  SLCHE-FH  (Geddie) 

Ft.  Hood,  TX  76544 
(817)  288-9917/21 
AV  738-9917/21 

Manned  System  Modeling  (Modeling) 

Mr.  James  Hartzell 
NASA-Ames  Research  Center 
MS  239-21 

Moffett  Field,  CA  94035 
(415)  694-5743 
AV  359-5743 

Sustained/Continuous  Operations  (Sus  Ops) 

LTC  Gerald  Krueger 

Walter  Reed  Army  Institute  of  Research  (WRAIR) 

ATTN:  SGRD-UWI -C/Krueger 

Washington,  DC  20307-5100 
(301)  427-5521 
AV  291-5521 

Tri-Service  Human  Factors  Standardization  Steering  Committee  (HFSSC) 
Mr.  Gerald  Chaikin 
Chief,  HEL  Detachment  -  MICOM 
SLCHE-MI  (Chaikin) 

US  Army  Missile  Command 
Redstone  Arsenal,  AL  35898-7290 
(205)  876-2048 
AV  746-2048 

Tri-Service  Workload  Coordinating  Committee  (Workload) 

Ms.  Heidi  Fiedler 

Naval  Underwater  Systems  Center 

Code  3512,  Bldg.  1171-1 

Newport,  RI  02841-5047 

(401)  841-2648 

AV  948-2648 

User-Computer  Interaction  (UCI) 

Director  USAHEL 
Bldg.  520 

ATTN:  SLCHE-CC  (Peterson) 

Aberdeen  Proving  Ground,  MD  21005-5001 
(301)  278-5957 
AV  298-5957 
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Voice-Interactive  Systems  (Voice) 
Ms.  Elaine  Schiller 
Naval  Ocean  Systems  Center 


Code  441 


San  Diego,  CA  92152 
(619)  225-7372 
AV  933-7372 


Committee  Chairs 


Human  Factors  Engineering  Guide  to  Systems  and  Equipment  Development 
(HFE-GSED) 

Dr.  Kenneth  Boff 

Aerospace  Medical  Research  Lab. 

AFAMRL/HEA 

Wright-Patterson  AFB,  OH  45433 
(513)  255-7596 
AV  785-7596 

Professional  Education,  Training,  and  Career  Development  (PETCD) 

Mr.  Todd  Jones 
U.S.  Coast  Guard 
G-DMT/54 

2100  Second  Street,  S.W. 

Washington,  DC  20593 
(202)  426-1058 


Technical  Society/Industry  Committee 
Dr.  Mark  M.  Brauer  (AIIE  Rep. ) 
Lockheed-Calif ornia  Co. 

P.0.  Box  551 
Burbank,  CA  91520-6951 
(818)  847-4345/1552 


( TS/I ) 


Liaison  Representatives 

Coast  Guard 

Mr.  Todd  Jones 
U.S.  Coast  Guard 
G-DMT/54 

2100  Second  Street,  S.W. 
Washington,  DC  20593 
(202)  426-1058 

Federal  Aviation  Administration 
TBD 


OUSDR&E  Proponent 

CAPT  Paul  R.  Chatelier 
OUSDR&E  ( R&AT ) 

Room  3D129  Pentagon 
Washington,  DC  20301 

•(202)  695-9777 
AV  225-9777 


Joint  Logistics  Commanders 
Joint  Technical  Coordinating  Group 
on  Human  Factors  Engineering  Human- 
Machine  Interface 

Dr.  James  C.  Geddie 
USAHEL  Field  Office 
HQ  TCATA 

ATTN:  SLCHE-FH  (Geddie) 

Ft.  Hood,  TX  76544 
(817)  288-9917/21 
AV  738-9917/21 

TAG  Coordinator 

Ms.  Louida  D.  Murray 
Eagle  Technology,  Inc. 

6714  W.  Geddes  Ave. 

Littleton,  CO  80123 
(303)  979-7441 
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ATTACHMENT  E.l 


CORRESPONDENCE 


DONALD  I .  CARTER 


OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


WASHINGTON  O  C  20301 


MfSf ARCH  ANO 
E  NGINftWING 

(R$AT) 


*2  OCf  1985 


MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  ARMY 

(RESEARCH,  DEVELOPMENT  AND  ACQUISITIONJ 
ASSISTANT  SECRETARY  OF  THE  NAVY 

(RESEARCH,  ENGINEERING  AND  SYSTEMS! 
ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 
(RESEARCH,  DEVELOPMENT  AND  LOGISTICS! 

SUBJECT:  Department  of  Defense  Human  Factors  Engineering 


Te 

chnical 

Ad 

visory  Group  (DoD  HFE 

TAG) 

i 

am  pi 

eased  at 

t 

he  response  of  the 

Ser 

vices  to  t 

he  Do 
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o  The  development  of  an  HFE  practitioner  training  course 
implemented  by  all  military  Services. 

o  The  cooperative  compilation  and  dissemination  of  research 
pertinent  to  the  hardware  and  system  design  process. 

o  The  regular  review  and  update  of  military  specifications, 
standards  and  design  documents  by  the  services  and  industry. 

I  encourage  the  Services  to  continue  their  vigorous  support 
of  the  HFE  TAG  and  other  similar  inter-Service  research 
coordinating  groups. 

Please  convey  our  gratitude  through  the  present  Service 
coordinators  for  the  HFE  TAG. 

Army :  Dr.  Michael  Strub,  Army  Research  Institute  for  the 

Behavioral  and  Social  Sciences 

Mr.  Clarence  Fry,  U.S.  Array  Human  Engineering 
Laboratory 

Navy :  Dr.  John  O'Hare,  Office  of  Naval  Research 

LCDR  Thomas  M.  Mitchell,  Pacific  Missile  Test  Center 

Air  Force:  Mr.  Cy  Crites,  Present  Chairman,  Edwards  Flight 
Test  Center 

Dr.  Richard  Schiffler,  Aeronautical  Systems  Division 


Donald  I.  Carter 

Acting  Deputy  Under  Secretary 

(Research  and  Advanced  Technology) 
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CORRESPONDENCE 


CAPT  PAUL  CHATELIER 


COCKP I T 


HFE  TOG 


fluTDHftTIDN  TECHNOLOGY  PROJECT 

Request  for  Assistance  from 

PEOUEST  (  f(X  J 

I  WOULD  APPRECIATE  THE  HELF  OF  THE  TAG  AND' OR  ONE  OR 

MORE  OF  ITS  Su»-TAGS  IN  OBTAINING  SOME  BACKGROUND  INFORMATION 

ON  HUMAN  FACTORS  RESEARCH  WHICH  MAY  HAVE  AN  IMPACT  OF  COCKPIT 

Automation  Technology.  The  information  could  come  as  answers  to  the 

FOLLOWING  GENERAL  QUESTIONS  WHICH  ARE  LISTED  IN  ORDER  OF  PRIORITY. 

1.  What  P  ♦  D  projects  are  under  hay  which  are  relevant  to  the 

DEVELOPMENT  AND  DESIGN  OF  COCKPIT  AUTOMATION?  WhAT  I  NEED  IS  A  LIST 
OF  TASKS  AND  PROJECTS  BY  PROGRAM  ELEMENT.  In  ADDITION*  POINTS  OF 
CONTACT  FOR  FOLLOW  -  UP  QUESTIONS  WOULD  BE  VERY  HELPFUL. 


P.  From  the  human  factors  perspective  what  are  the  issues* 

WHAT  ARE  THE  NON  ISSUES.  I.E.  WHAT  DO  WE  KNOW  f  WHAT  DON'T  WE  KNOW  THAT 
IS  RELEVANT  TO  THE  DEVELOPMENT  AND  DESIGN  OF  COCKPIT  AUTOMATION.  FOR 
EXAMPLE »  HAVE  WE  MADE  SIGNIFICANT  PROGRESS  OVER  THE  PAST  FEW  YEARS  IN 
DEVELOPING  WAYS  TO  ALLOCATE  FUNCTIONS  BETWEEN  CREWMEMBER  AND  MACHINE. 

Rre  any  new  specifications  available  or  emerging  on  when  and  how  much 

to  AID  HUMAN  DECISION  MAKING?  HERE »  I  WOULD  LIKE  TO  OBTAIN  A  LIST  OF 
ISSUES/  NON  ISSUES  AND  RELATED  SUMMARIES  OF  R  +  D  PROGRESS. 


3.  What  thoughts* 

TRADE-OFF  BETWEEN 


OR  INSIGHTS  DOES  THE  TftG 
TRAINING  AND  AUTOMATION? 


HAVE 


ABOUT 


THE 


To 

BY 


BE  FULLY  HELPFUL  *  THE  INFORMATION  WOULD 
THE  FIRST  WEEK  IN  JANUARY  1986. 


NEED 


TO  BE  AVAILABLE 
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BACKGROUND 

In  April  1985?  a  topicw.  reviem  on  Cockpit  Automation  Technology  mas 

HELD  AT  THE  REBUEST  OF  THE  DEPUTY  UNDER  SECPETAAY  OF  DEFENSE  FOR 

Research  and  Advanced  Technology.  The  reviem  dreh  84  people  from  nine 
organizations.  The  purpose  df  the  reviem  has  to  exchange  inforhation 

AND  IDENTIFY  OPPORTUNITIES  FOR  FUTURE  DEVELOPMENT. 

R  NUMBER  OF  CONCLUSIONS  EMERGED  FROM  THO  DAYS  OF  PRESENTATIONS  AND 

discussions.  These  are  summarized  in  Attachment  2.  The  major 

CONCLUSION?  HOWEVER?  CALLED  FDR  THE  PREPARATION  OF  COORDINATED? 
INTER-SERVICE  POLICY  AND  PLANS  FDR  COCKPIT  AUTOMATION  TECHNOLOGY  R  &D. 

In  response  to  that  conclusion  the  Deputy  Under  Secretary  asked  the 
Service  Secretaries  fdr  Research  to  task  their  JDL  to  undertake  the 

RECOMMENDED  PLANNING  (ATTACHMENT  1>.  THE  JDL  ARE  BEING  ASKED  TO 
PROVIDE  DRAFT  PLANS  BY  JANUARY?  1986?  AND  A  FINAL  STATEMENT  BY  APRIL. 

I  MAY  BE  CALLED  UPON  TO  PROVIDE  SOME  ASSISTANCE  AND  MOULD  LIKE  TO  HAVE 
ENOUGH  BACKGROUND  INFORMATION  TO  DO  SO. 

SDME  SUGGESTIONS  FDR  WHAT  TD  INCLUDE  IN  THE  INFORMATION 

Question  1.  A  listing  of  current  R  ♦  D  tasks  or  mork  unit  by  Program 
Element  and  Project  Number?  hith  a  very  brief  description  of  the 

EFFORT  MOULD  BE  HELPFUL. 

Question  2.  Conclusions  and  recommendations  fpom  a  1982  report  entitled 
"Automation  in  Combat  Aircraft"?  National  Academy  Press?  Washington? 

D.C.?  MIGHT  PROVIDE  A  USEFUL  POINT  OF  DEPARTURE  IN  ADDRESSING  THIS 

ruESTiON.  I've  provided  a  list  of  those  conclusions  and  recommendations 
tn  Attachment  3.  For  example?  are  the  issues  implicit  in  that  list 
'TILL  VALID.  Has  THEIR  IMPORTANCE  -  AS  PROBLEMS  -  DIMINISHED  BECAUSE 
OF  PROGRESS  IN  HUMAN  FACTORS  OR  ENGINEERING  TECHNOLOGY?  HAS  IT 
INCREASED  BECAUSE  OF  SIDE  EFFECTS  OR  LED  TO  NEM  ISSUES7  THE  LIST  IS 
NOT  MEANT  TO  BE  CONSTRAINING  BUT  RATHER  IS  AN  ILLUSTRATIVE  POINT  OF 
DEPARTURE. 

Questions  3  and  4  are  open-ended.  Any  suggestions?  ideas  or  information 

WOULD  BE  HELPFUL. 
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WASHINGTON  DC  20)01 


RESEARCH  AND 


1 6  SEP  1985 


engineering 


(RSAT) 

MEMORANDUM  FOR  DIRECTOR,  DEFENSE  ADVANCED  RESEARCH  PROJECT  AGENCY 
SUBJECT:  Combat  Aircraft  Cockpit  Automation  Technology 


Enclosed  is  a  summary  report  on  the  DoD/NASA  Topical  Review 
of  Combat  Aircraft  Cockpit  Automation  technology  held  24- 
25  April  1985  at  the  Naval  Air  Development  Center,  Warminster, 
Pennsylvania.  Enclosed  also  is  a  memorandum  I  have  sent  to 
Assistant  Secretaries  of  the  three  SerVices  requesting  their 
support  of  a  joint,  continuing  coordination  function  in  this 
field,  to  implement  a  major  recommendation  of  the  report. 

Because  DARPA  is  intimately  concerned  with  this  general  area 
of  research  and  technology,  it  is  vital  to  the  success  of  the 
planning  activities  that  you  participate  in  them.  It  would  be 
appreciated  if  you  would  inform  Mr.  Raymond  F.  Siewert,  my  staff 
point  of  contact,  telephone  697-7922,  of  your  designee.  Thank 
you  for  your  cooperation. 


Donald  I .  Carter 

Acting  Deputy  Under  Secretary 

(Research  and  Advanced  Technology) 


Enclosures  (2) 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


RESEARCH  AND 
ENGINEERING 

(RSAT) 


WASHINGTON  DC  20101 


1  6  SEP  1985 


Dr.  Raymond  Colladay 

Associate  Administrator  for  Aeronautics 
and  Space  Technology 

National  Aeronautics  and  Space  Administration 
Washington,  DC  20546 

Dear  Dr.  Colladay: 

Enclosed  is  a  summary  report  on  the  DoD/NASA  Topical  Review 
of  Combat  Aircraft  Cockpit  Automation  technology  held 
24-25  April  1985  at  the  Naval  Air  Development  Center,  Warminster, 
Pennsylvania.  Enclosed  also  is  a  memorandum  I  have  sent  to 
Assistant  Secretaries  of  the  three  Services  requesting  their 
support  of  a  joint,  continuing  coordination  function  in  this 
field,  to  implement  a  major  recommendation  of  the  report. 

Because  NASA's  research  and  technology  in  many  of  the 
fundamental  disciplines  and  in  system  integration  are  importantly 
related  to  our  objectives,  I  invite  NASA's  participation  in 
planning  and  implementing  the  coordination  we  desire.  To  assist 
us,  it  would  be  helpful  if  you  would  designate  a  knowledgeable, 
senior  member  of  your  staff  to  work  with  us,  and  notify 
Mr.  Raymond  F.  Siewert,  my  staff  point  of  contact.  He  can  be 
reached  at  telephone  number  697-7922. 

Sincerely, 


Acting  Deputy  Under  Secretary 
(Research  and  Advanced  Technology) 


Enclosures  (2) 
a/s 
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OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


ENGINEERING 


(RSAT) 


WASHINGTON  DC  70101 


1  6  SEP  1985 


MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  ARMY  (RESEARCH,  DEVELOPMENT 

AND  ACQUISITION) 

ASSISTANT  SECRETARY  OF  THE  NAVY  (RESEARCH,  ENGINEERING 
AND  SYSTEMS) 

ASSISTANT  SECRETARY  OF  THE  AIR  FORCE  (RESEARCH, 
DEVELOPMENT  AND  LOGISTICS) 


SUBJECT:  Combat  Aircraft  Cockpit  Automation  Technology 


Enclosed  is  a  summary  report  on  a  Topical  Review  of  combat 
aircraft  cockpit  automation  technology  held  24-25  April  1985  at 
the  Naval  Air  Development  Center,  Warminster,  Pennsylvania.  All 
three  Services,  DARPA  and  NASA  participated  in  the  review. 


The  primary  objectives  were  to  provide  a  broad  exchange  of 
information  rather  than  technology  details,  and  to  provide  a 
forum  to  discuss  issues  common  to  all  the  Services  as  they  move 
toward  increased  automation  in  combat  aircraft.  All  the 
conclusions  and  recommendations  in  the  report  should  be  of 
interest  to  you,  but  I  want  to  draw  your  attention  specifically 
to  the  final  recommendation,  which  is  that  coordinated  plans 
should  be  provided  for  developing  the  basic  technologies  and 
system  design  methodologies. 


The  importance  of  this  recommendation  suggests  to  me  that  it 
should  be  implemented  through  the  Joint  Directors  of 
Laboratories,  with  participation  by  each  of  the  Services,  DARPA 
and  NASA.  Accordingly,  I  request  that  the  JDL  be  tasked  to 
produce  a  coordinated  technology  development  plan  as  well  as  a 
plan  for  tracking  and  coordinating  its  execution.  It  is 
requested  that  both  plans  be  provided  within  six  months  of  this 
date,  with  a  status  report  at  mid-point. 


I  nave  asked  Mr.  Raymond  F.  Siewert,  Director  of  ray  Military 
Systems  Technology  Office,  to  be  the  principal  point  of  contact 
for  this  task  in  my  office.  He  is  available  to  clarify  any 
questions  that  may  arise  and  to  assist  the  JDL  in  the  timely 
development  of  the  plans.  His  telephone  number  is  (202)  697-7922. 


Enclosure 


Donald  I.  Carter 

Acting  Deputy  Under  Secretary 

(Research  and  Advanced  Technology) 


SUMMARY  REPORT:  TOPICAL  REVIEW 

OF  COMBAT  AIRCRAFT  COCKPIT 
AUTOMATION  TECHNOLOGY 


I.  INTRODUCTION 

A  Topical  Review  of  Combat  Aircraft  Cockpit  Automation 
(CACA)  technology  took  place  on  April  24-25,  1985  at  the  Naval 
Air  Development  Center,  Warminster,  Pennsylvania.  More  than 
eighty  persons,  largely  from  the  Military  Services,  Defense 
Agencies  and  NASA  (see  enclosure  1)  participated  in  the  Review. 

A  Topical  Review  of  this  area  had  been  requested  by  the 
Deputy  Under  Secretary  of  Defense,  Research  and  Engineering 
(Research  and  Advanced  Technology)  because  of  its  increasing 
importance  and  technical  sophistication,  and  due  to  the  rapidity 
with  which  each  of  the  Services  is  proceeding,  generally  inde¬ 
pendently. 

The  primary  objective  of  the  Review  was  information  exchange 
and  identification  of  issues  and  opportunities.  Accordingly, 
most  program  presentations  and  discussions  of  issues  were  largely 
summary  in  nature,  and  neither  the  technical  nor  programmatic 
details  were  covered  in  depth.  The  first  day  of  the  Review  (see 
agenda,  enclosure  2)  was  devoted  to  presentations  by  all  partici¬ 
pating  organizations,  and  the  second  day  was  spent  on  a  series  of 
panel  discussions  with  audience  participation.  During 
preparatory  meetings,  each  Panel  formulated  an  Issue  Paper  to 
focus  its  views.  These  Issue  Papers  are  available  on  request  to 
the  Office  of  Military  Systems  Technology,  OUSDRE(R^AT) . 

The  Findings  and  Conclusions,  and  Recommendations  from  this 
Topical  Review  are  contained  in  Sections  II  and  III  of  this 
Summary  Report.  Overall,  it  is  believed  that  the  Review  success¬ 
fully  achieved  its  objectives  and  was  considered  valuable  by  most 
participants  and  attendees. 

II.  FINDINGS  AND  CONCLUSIONS 

These  findings  and  conclusions  reflect  views  of  a  group  of 
CACA  Steering  Group  members  (or  their  designees)  and  several 
Panel  Chairmen. 

1.  Cockpit  automation  is  an  intensively  interdisciplinary 
field,  but  research  activities  in  many  areas  are  not  well  coordi¬ 
nated  . 
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Combat  aircraft  cockpit  design  integrates  the  entire  weapon 
system  and  includes  all  technologies  related  to  aircraft  and 
weapon  subsystem  functions  and  crew/machine  interfaces.  It  also 
includes  disciplines  concerned  with  human  physical  and  mental 
capabilities  and  limitations,  and  the  emerging  disciplines 
related  to  expert  systems  and  machine  (artificial)  intelligence 
applications  to  combat  aircraft  missions  and  survival.  Although 
research  is  proceeding  in  most  of  these  areas,  the  Topical  Review 
did  not  reveal  any  overall  coordination,  plans  or  roadmaps  for 
cockpit  automation  research  and  technology  development  and  trans¬ 
ition. 


2.  Standards  and  methods  for  cockpit  design  are  lacking 
today,  and  current  practices  are  "bottom-up"  rather  than  "top- 
down  . " 


The  design  approach  for  each  new  aircraft  cockpit  (that  is, 
for  total  system  integration,  control  and  management)  has  become 
quite  complicated  and  time-consuming.  It  is  performed  primarily 
in  an  ad  hoc  manner  in  every  new  situation  and  is  more  heavily 
influenced  by  dominant  personal  views,  experience  and  personality 
factors  than  by  established  criteria  or  procedures.  Each  new 
aircraft  cockpit  system,  faced  with  new  requirements  to  be 
accommodated  and  new  technologies  to  be  incorporated,  evolves 
largely  through  a  trial-and-error ,  "bottora-up"  rather  than  "top- 
down"  process,  without  an  established  set  of  criteria  and 
integration  principles.  While  this  fact  may  reflect  the  newness 
and  rapid  evolution  of  requirements  and  technology,  it  also 
suggests  the  need  for  maturation  to  capitalize  both  on  both 
experience  and  broad  systems  analysis. 

3.  The  future  role  of  man  in  the  combat  aircraft  cockpit  is 
not  well  defined. 

Cockpit  automation  (transferring  functions  once  performed  by 
humans  to  the  machine)  is  justified  to  the  degree  it  can  improve 
the  probability  of  increased  mission  effectiveness  or  surviva¬ 
bility.  Aircrews  presently  provide  essential  system  capabilities 
involving  comprehension,  reasoning  and  evaluation,  knowledge, 
skill,  "judgment"  and  decision  making,  as  well  as  overall  system 
management.  But  as  technologies  advance  in  avionic  and  computer- 
based  disciplines,  including  expert  systems  and  machine  intelli¬ 
gence  ,  some  (perhaps  many)  of  those  human  capabilities  may  become 
future  candidates  for  automation. 

While  the  optimal  role  of  man  and  his  relationship  to  the 
rest  of  a  system  incorporating  those  future  technological 
advances  is  at  the  moment  unclear,  most  conferees  believed  that 
man  should  somehow  remain  "in  charge".  How  this  arrangement  was 
to  be  accomplished  remained  undecided.  Most  agreed  that  human 
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capabilities  involving  imagination,  intuition  and  innovation 
should  remain  available,  and  that  man  must  have  a  "last  word"  or 
ultimate  decision  and  override  capability,  particularly  in 
unusual  and  emergency  situations  where  "judgment"  should  be 
exercised.  Man  should  have  the  option  of  breaking  the  rules  or 
exceeding  allowable  machine-set  limits,  even  in  apparent  contra¬ 
diction  to  his  self  interest,  and  of  making  "illogical"  or 
"wrong"  decisions  if  his  experience  or  intuition  should  indicate 
that  such  desperate  actions  might  be  preferred  choices. 

Notwithstanding,  it  was  agreed  that  in  the  future  man  can't 
do  it  all:  that  the  situation  will  be  too  complex,  fast  moving 
and  incomprehensible  for  man  alone.  Therefore,  it  was  evident 
that  a  new  symbiotic  relationship  between  man  and  intelligent 
machine,  not  yet  defined,  will  have  to  evolve. 

4.  Appropriate  measures  of  effectiveness  will  be  needed  to 
help  guide  future  automation  choices  and  decisions  for  combat 
aircraft . 

Before  correct  choices  can  be  made  on  future  cockpit 
automation,  better  techniques  will  be  needed  to  anticipate  and 
assess  their  actual  effects  and  utility.  Crit.eria  for 
performance  and  effectiveness  evaluation  have  not  been 
established  except  perhaps  in  a  philosophical  sense.  More 
specific  criteria  must  be  developed  concurrently  with  the 
development  of  the  new  technologies  to  which  they  will  be 
applied,  as  well  as  valid  means  for  anticipating  and  assessing 
the  effect  and  utility  of  any  proposed  new  automation. 

III.  RECOMMENDATIONS 

1.  The  development  of  critical  technologies  should  be 
continued . 

Research  in  each  area  critical  to  the  development  of  future 
combat  aircraft  cockpit  automation  technology  should  proceed  with 
high  priority  in  each  user  service  and  contributing  research 
organization.  One  such  area  includes  all  aspects  of  the 
man/machine  interface  for  better  communication  and  interaction, 
including  additional  research  on  voice  recognition/interaction. 

A  second  area  includes  expert  systems  and  machine  intelligence 
based  on  advances  in  every  area  of  computer  science,  including 
"knowledge  engineering"  to  organize,  correlate,  and  make 
information  bases  accessible  in  a  dynamic  environment.  Another 
critical  area  for  research  is  that  of  human  characteristics, 
capabilities  and  limitations  in  the  combat  aircraft  environment. 

2.  A  mature,  systems  approach  to  combat  aircraft  cockpit 
design  and  integration  should  be  developed. 
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The  time  has  come  when  the  complexity  of  the  task  demands  a 
systems  approach  instead  of  a  Mbottom-upM ,  trial-and-error , 
personality-dominated  approach.  Deriving  maximum  benefits  from 
both  human  and  machine  characteristics  is  virtually  impossible 
otherwise,  and  the  penalties  for  failing  to  deal  with  the  whole 
set  of  requirements  and  available  technologies  as  a  system  would 
include  wasted  time  and  expense,  undesirable  compromise  of 
capabilities,  and  great  difficulties  (perhaps  impossibilities)  in 
introducing  necessary  corrections  or  desirable  improvements  down¬ 
stream.  Interdisciplinary  in  nature,  the  development  of  system 
design  methodologies  and  design  aids  should  reflect  the  expertise 
of  organizations  familiar  with  large,  complex  systems. 

3.  The  optimal  role  of  man  in  a  new  man-intelligent  machine 
symbiosis  must  be  established. 

A  total  systems  approach  for  highly  automated  future  combat 
aircraft  design  integration  is  needed  to  establish  the  optimal 
role  of  man.  The  degree  and  circumstances  for  his  management  of 
information,  decision  making,  supervision  and  control  must  be 
established  when  there  are  counterpart  machine  capabilities 
available.  Situationally  dependent  and  override  options  for 
every  phase  of  the  mission  need  resolution  to  preserve  maximum 
desirable  flexibility. 

4.  Coordinated  plans  should  be  provided  for  developing  the 
basic  technologies  and  system  design  methodologies  for  future 
combat  aircraft  cockpit  design  and  integration. 

The  magnitude  and  importance  of  the  subject,  the  uneven 
evolution  of  the  underlying  technologies,  and  the  requirements 
for  a  systems  approach  to  methodology  indicate  the  need  for 
greater  coordination  and  overall  planning  of  research  and 
transition  opportunities.  No  conclusion  was  reached  on  how  this 
might  best  be  accomplished.  Options  include  coordination  at  the 
Joint  Directors  of  Laboratories  level,  or  within  the  Office  of 
the  Secretary  of  Defense,  or  by  an  ad  hoc  interorgani zational 
group  appointed  to  work  toward  common  goals.  An  important 
requirement  of  any  coordinating  group  is  that  the  constituent 
members  be  influential  within  .their  own  organizations.  Another 
important  requirement  is  that  the  coordination  group  include 
combat  pilots  and  crew  whose  experience  and  insights  are 
invaluable . 
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SUMMARY 


This  report  is  concerned  with  the  automation  of  combat  aircraft 
to  support  the  mission  of  the  United  States  Air  Force.  According  to 
the  statement  of  task  decided  upon  by  the  Air  Force  and  the  National 
Research  Council,  this  study  addresses  the  issue  in  terms  of  manned 
systems.  Given  the  limited  time  available  for  the  study,  the  C: 
mittee  on  Automation  of  Combat  Aircraft  concentrated  on  single-sea . 
fighters. 

The  term  "automation"  is  subject  to  diverse  interpretations.  It 
has  been  used  variously  to  describe  the  control  of  a  single  function 
by  a  simple  on-off  mechanism,  as  in  the  thermostatic  control  of 
building  temperatures.  It  has  been  used  to  describe  the  concurrent 
display  of  data  from  various  sources  to  a  person  for  interpreta¬ 
tion:  an  example  is  the  cockpit  displays  of  a  modern  fighter  air¬ 
craft,  which  combine  information,  for  instance,  from  a  radar  and  an 
electro-optical  sensor  with  computer-derived  flight  parameters. 
Automation  has  also  been  used  to  describe  the  control  of  complex 
processes,  in  which  the  automated  system  replaces  some  human  moni¬ 
toring,  decisionmaking,  and/or  motor  functions,  as  in  automatic 
flight  control. 

In  this  study,  the  committee  defined  automation  as  those  pro¬ 
cesses  by  which  essential  functions  can  be  performed  with  partial, 
intermittent,  or  no  intervention  by  the  pilot.  In  this  report,  the 
term  automation  will  describe  any  effort  to  move  the  cognitive  pro¬ 
cesses  of  flying  the  aircraft  and  managing  its  weapons  from  the 
pilot  or  aircrew  to  a  computer-dominated  system. 


Kiman  Limitations  in  the  Combat  Environment 

Technological  advances  in  the  past  two  decades  have  made  possi¬ 
ble  the  development  of  complex  and  more  competent  aircraft  that  can 
fly  under  more  difficult  conditions  (such  as  close  to  the  ground,  at 
night,  or  underneath  the  weather),  and  at  faster  speeds.  They  can 
also  perform  more  complex  missions,  such  as  simultaneously  attacking 
multiple  targets. 

At  the  same  time,  and  perhaps  as  a  consequence  of  these  techno¬ 
logical  advances,  the  environment  in  which  aircraft  must  fly  and 
fight  has  become  more  dangerous.  Threats  from  the  air  and  ground 
are  faster  moving  and  harder  to  detect.  Not  only  are  ihe  aircraft 
vulnerable  to  enemy  attack,  but  they  can  also  be  shot  down  acciden¬ 
tally  by  U.S.  surface-to-air  missiles. 
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The  only  element  that  has  not  changed  significantly  over  the 
years  is  the  human  operator.  The  pilot  is  limited  in  his  ability  to 
assimilate  and  perform  tasks.  He  may  not  be  able  to  handle  the  in¬ 
creased  workload  involved  in  operating  today's  faster,  more  highly 
mechanized  aircraft.  Limitations  in  human  capabilities  are  diffi¬ 
cult  to  overcome,  and  as  yet  have  not  been  completely  described. 
The  proper  use  of  automation  in  aircraft  could  help  to  overcome 
these  limitations. 

In  fact,  both  the  military  and  commercial  aviation  communities 
have  been  using  automation  to  various  degrees  over  the  years.  Ex¬ 
tensive  automation  has  been  used  in  manned  vehicles  such  as  the 
Space  Shuttle.  Air  Force  aircraft  such  as  the  F-106,  F/FB-111,  and 
others  are  also  capable  of  semi-automated  flight.  Yet  automation 
presents  its  own  difficulties,  in  terms  of  cost,  reliability,  main¬ 
tenance,  and  complexity  of  operation.  To  reduce  cost  and  complex¬ 
ity,  and  because  of  technology  limitations,  the  most  recent  Air 
Force  aircraft  (F-15,  F-16,  and  A-10)  have  been  designed  with  only 
selected  automated  features,  to  perform  essential  functions.  These 
aircraft  can  be  considered  semi-autcmated  at  best. 

Operating  these  high-performance  aircraft  (the  F-15,  F-16,  and 
A-10)  places  great  demands  on  the  pilot.  The  increased  workload 
that  results  from  the  complexity  of  the  aircraft  and  the  challenge 
of  today's  combat  situation  may  lead  to  errors  or  mission  failure. 

This  workload  is  especially  evident  in  the  cockpit.  Recent  ad¬ 
vances  in  avionics  and  weapons  technology  have  led  to  an  exponential 
increase  in  the  numbers  of  cockpit  displays  and  controls  since  the 
1920s.  Through  these  displays,  the  pilot  is  given  large  quantities 
of  information  in  rapid  succession.  He  needs  to  be  aware  of  the 
aircraft's  internal  and  external  situation,  in  addition  to  operating 
the  fire-control  system,  selecting  and  firing  munitions,  and  posi¬ 
tioning  a  large  nunber  of  switches.  When  the  pilot  focuses  his 
attention  on  a  particularly  demanding  function,  such  as  locking  onto 
an  enemy  target  or  following  rugged,  hilly  terrain,  he  may  lose 
track  of  others,  even  when  provided  with  some  automated  functions 
(s^ch  as  weight  distribution,  fuel-flow  management,  flight -attitude 
control,  and  threat  warning). 

In  the  past,  when  missions  and  aircraft  were  simpler,  the  pilot 
was  able  to  assess  the  current  situation  and  draw  on  past  experience 
in  executing  his  mission.  If  the  trend  toward  complexity  of  air¬ 
craft  and  combat  situations  continues,  the  pilot  will  be  increas¬ 
ingly  unable  to  perform  his  mission  without  extensive  aid. 

By  the  1990s,  combat  aircraft  could  be  more  fully  automated  to 
aid  and  support  the  pilot  in  performing  his  tasks  and  keeping  track 
of  the  overall  situation,  thereby  increasing  the  probability  that 
the  mission  will  succeed  and  the  pilot  and  aircraft  will  survive. 

yrt 


E.  2-16 


If  automation  is  truly  to  improve  mission  performance,  aircraft 
designers  must  carefully  consider  where  automation  would  best  serve 
pilots’ •  needs.  They  must  examine  not  only  the  technology,  but  the 
human  factors  involved.  This  will  require  an  understanding  of  how 
pilots  process  and  assimilate  information  and  how  they  think  about 
their  tasks,  as  well  as  an  understanding  of  the  performance  charac¬ 
teristics  of  the  controls  and  displays  through  which  the  pilot  and 
the  automated  systems  interact. 


New  Opportunities  for  Automation 

The  committee  believes  that  new  Air  Force  development  and  appli¬ 
cation  of  automated  features  based  on  computer  technology  can  im¬ 
prove  the  operational  effectiveness  of  combat  aircraft.  In  addi¬ 
tion,  it  will  enhance  the  thances  of  survival  for  both  the  aircraft 
and  the  pilot.  The  reliability  and  capacities  of  computers  are  in¬ 
creasing.  At  the  same  time,  size,  weight,  and  cost  per  unit  of  com¬ 
puter  power  is  decreasing  enough  to  Justify  its  extensive  use  to 
provide  a  comprehensive,  integrated,  up-to-the-minute  model  of  the 
aircraft  and  its  mission.  Also,  since  software  costs  are  dominating 
hardware  costs,  it  makes  economic  sense  to  design  systems  with  flex¬ 
ibility  to  accommodate  new  requirements  and  developments  over  the 
operational  life  of  the  aircraft. 


Acceptance  of  Automation 

Air  Force  personnel  are  generally  receptive  to  further  automa¬ 
tion  of  combat  aircraft.  Although  senior  managers  appreciate  the 
potential  of  automation,  they  are  not  oversold  on  its  merits. 
Because  of  their  concern  about  increased  cost  and  complexity  of 
automated  systems,  and  the  possibility  of  low  reliability,  managers 
are  conservative  in  making  commitments  to  the  increased  use  of  auto¬ 
mated  systems. 

The  aircrews  exhibit  similar  attitudes,  with  a  strong  "show  me" 
tone.  Pilots  wish  to  retain  the  ability  to  select  automated  func¬ 
tions  and  to  intervene  in  their  operation.  Aircrews  want  simplicity 
in  execution  as  well  as  performance.  They  accept  automation  of 
functions  that  humans  cannot  perform  adetjjately,  functions  that  dis¬ 
tract  pilots  from  critical  tasks,  and  functions  or  routines  that  are 
infrequently  performed  and  can  be  done  more  reliably  through  automa¬ 
tion. 

Emphasis  in  the  development  of  automated  systems  has  been  on 
data  and  information  displays  and  on  sensors— the  "outer"  ends  of 
aircraft  automation.  The  "inner"  part  of  the  problems— processing 
these  inputs  from  diverse  information  sources  to  improve  pilot 
awareness  of  the  outside  world  and  the  status  of  his  aircraft— is 
Just  beginning  to  receive  needed  attention. 
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Several  current  programs  are  developing  technology  that  can  help 
to  identify  the  operational  approach  to  and  need  for  automation. 
These  programs  include  the  Advanced  Fighter  Technology  Integrator/ 
Advanced  Maneuvering  Attack  Systems  (AFTI-16/AMAS)  program,  the  Low 
Altitude  Navigation  Targeting  Infrared  for  Night  (LANTIRN)  program, 
and  development  of  terrain  following  and  terrain  avoidance  (TF/TA) 
technology.  (See  the  Glossary  for  definitions  of  these  programs.) 
The  prototypes  developed  in  these  programs  can  contribute  to  under¬ 
standing,  and  perhaps  to  satisfying,  the  operational  community’s 
immediate  stated  need  for  improved  capabilities  to  fly  low,  at  night, 
and  during  severe  weather,  using  terrain  for  cover.  The  long-range 
goal  is  the  ability  to  fly  missions  effectively  in  all  types  of 
weather,  in  the  daytime  and  at  night,  devastating  the  enemy  and  sur¬ 
viving  in  hostile  environments.  To  realize  this  goal,  more  complete 
forms  of  automation  that  do  not  detract  from  aircraft  availability 
(because  of  "downtime"  or  problems  with  reliability)  will  be  neces¬ 
sary. 


Questions  of  Reliability 

The  availability  of  combat  aircraft  is  a  central  concern.  It 
depends  on  the  reliability  of  the  aircraft  and  their  subsystems,  and 
on  the  adequacy  of  maintenance  and  logistics.  It  is  reasonably  clear 
that  reliability  would  be  improved  if  these  subsystems  were  subjected 
to  follow-on  cycles  of  modification  after  feasibility  demonstration 
and  testing. 

The  committee  has  not  examined  the  problem  of  unreliability  in 
enough  depth  to  ascertain  its  actual  causes,  but  inadequacies  in 
testing,  design  margins,  packaging,  vibration  isolation,  and  tempera¬ 
ture  control,  along  with  excessive  handling,  appear  to  be  sources  of 
trouble.  Developers  of  electronic  systems  for  both  commercial  air¬ 
craft  and  strategic  missiles  have  concentrated  on  these  items  in 
their  programs  to  attain  high  reliability.  We  do  know  that  environ¬ 
mental  factors  greatly  influence  the  reliability  of  aircraft  equip¬ 
ment,  and  that  a  tactical  aircraft  is  exposed  to  far  more  hostile 
conditions  than  a  commercial  aircraft  or  a  missile  in  a  silo.  These 
conditions  include  frequent  power  on-off  cycles,  short  but  intense 
sorties,  and  inexperienced  maintenance  crews. 

Addressing  some  of  these  problems  will  help  to  improve  aircraft 
reliability,  especially  if  combined  with  designs  that  provide  redun¬ 
dancy  of  components  and  fault  isolation  and  detection.  In  addition, 
an  overall  systems  architecture  for  the  aircraft  that  allows  the 
integration  of  various  components  and  accommodates  new  developments 
will  be  of  great  benefit  in  developing  reliable  aircraft.  Finally, 
pilots  demand  provisions  to  intervene  in  the  case  of  subsystem 
failure. 
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The  Need  for  Integrated  Systems 


Technology  is  available  for  developing  suitable  automation  for 
combat  aircraft  of  the  1990s,  and  to  modify  some  existing  aircraft 
and  some  now  in  production.  To  achieve  this  goal,  the  Air  Force  will 
have  to  define  the  requirements  for  automation,  and  recognize  that 
the  task  requires  an  integrated  systems  approach,  rather  than  a  col¬ 
lection  of  piecemeal  efforts.  For  such  an  integrated  approach,  a 
focal  point  for  automation  should  be  established.  Flight  trajectory 
and  attitude  control  should  be  the  focal  point  for  all  aircraft  auto¬ 
mation.  This  function— the  maintenance  of  the  correct  flight  trajec¬ 
tory  and  attitude,  or  orientation  of  the  plan,  over  time — is  the 
heart,  mind,  and  nerves  of  the  aircraft,  to  which  all  other  functions 
are  keyed.  Therefore,  it  can  form  the  core  of  a  truly  integrated 
system  of  automated  features. 

A  flexible  architecture  for  flight  trajectory  and  attitude  con¬ 
trol  Is  vital:  logical  partitioning  and  standard  interfaces  will  be 
necessary  to  permit  the  later  addition  of  sensors  and  weapons  for 
more  highly  developed  systems.  This  is  a  prudent  way  to  accommodate 
change  in  a  fleet  of  aircraft  that  will  remain  in  use  for  a  long 
time.  Correct  core  design  can  lead  to  standardized  software  and 
hardware  applicable  to  more  than  one  airplane.  A  deliberate  systems- 
oriented  approach  is  essential  to  avoid  a  proliferation  of  incompat¬ 
ible  functions,  controls,  and  displays.  A  basic  model  should  be 
developed  for  the  architectural  design.  This  model  should  include, 
at  the  least,  the  specifications  of  cockpit  features,  flight-  and 
engine -control  systems,  and  navigation  elements. 

A  comprehensive,  thorough,  adequately  supported  testing  and  eval¬ 
uation  program  must  be  a  major  element  of  any  such  development.  The 
operational  and  engineering  flight-test  communities  should  be  in¬ 
volved.  If  effective  automated  systems  are  to  be  available  in  the- 
1990s,  now  is  the  time  to  start. 


A  Note  on  Safety 

This  study  group's  discussions  with  military  pilots  lead  it  to 
conclude  that  flying  high-performance  aircraft  is  hazardous,  and  is 
likely  to  be  more  so  in  the  future.  During  the  course  of  this  study, 
the  group  received  information  on  certain  safety-related  technologies 
and  systems  that  will  affect  future  Air  Force  research  and  develop¬ 
ment  (R&D).  While  these  technologies  and  systems  are  not  strictly 
within  the  scope  of  the  study,  they  are  of  such  importance  that  com¬ 
ments  are  appropriate. 
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Identification,  Friend,  Foe,  or  Neutral 

A  longstanding  problem  that  still  has  no  clear  solution  is  the 
identification  of  friends,  foes,  and  neutral  forces,  abbreviated 
IFFN.  Reliable  means  of  identifying  friendly,  as  well  as  enemy, 
forces  ere  vital.  The  combat  arena  of  the  future  will  be  even 
faster  moving  than  that  of  today  and  will  be  filled  with  intense 
activity.  If  identifications  must  be  made  visually  by  pilots,  a 
significant  tactical  advantage  of  long-range  detection  sensors  and 
long-range  weapons  will  be  lost.  The  development  and  use  of  reli¬ 
able  equipment  for  the  identification  of  friendly,  hostile,  or  neu¬ 
tral  forces  is  crucial  to  increasing  combat  effectiveness  and  the 
chances  of  survival.  One  possible  solution  may  involve  the  coupling 
of  several  sensors  and  systems  to  supply  intelligence  and  positional 
information,  as  well  as  information  on  the  situation  external  to  the 
aircraft. 

The  problems  of  target  assignment  and  acquisition  go  along  with 
that  of  identification.  Once  identification  is  made,  the  next  prob¬ 
lem  is  the  efficient  use  of  the  force  to  attack  multiple  targets. 
When  there  are  multiple  targets,  it  is  helpful  to  have  a  system  for 
exchanging  information  among  individual  aircraft  to  ensure  that  each 
plane  attacks  a  different  target.  Such  a  target-assignment  process 
would  make  more  effective  use  of  the  fighting  force  and  would  also 
improve  the  pilots'  general  awareness  of  combat  situations.  Such  a 
process  is  possible  through  the  use  of  systems  such  as  the  multiple- 
access  digital  data  link  known  as  the  Joint  Tactical  Information 
Distribution  System  (JTIDS).  This  and  similar  systems  could  aid  in 
the  identification  process,  and  could  have  the  ability  to  transmit 
large  quantities  of  situational  intelligence  data. 


Unmanned  Vehicles 

Although  this  study  concentrated  on  manned  aircraft,  the  use  of 
fully  automated  unmanned  vehicles  (UVs)  is  worthy  of  comment.  In 
some  combat  situations,  such  aircraft  can  be  effective,  and  they 
reduce  the  loss  of  pilots.  The  technology  is  available  to  develop 
UVs  that  can  perform  automatically,  without  human  intervention,  such 
missions  as  defense  suppression,  reconnaissance,  surveillance,  and 
communications  relay.  Technology  now  under  development  for  both 
manned  and  unmanned  vehicles  should  allow  the  automatic  performance 
of  electronic  warfare,  damage  assessment,  and  target  assignment. 
The  increasing  power  of  small  data  processing  machines  has  implica¬ 
tions  for  the  development  of  UVs,  as  do  advances  in  sensors,  struc¬ 
tural  materials,  and  propulsion  technology.  Given  these  develop¬ 
ments,  the  prospects  are  increasingly  good  for  including  UVs  8S  a 
part  of  a  mixed  force  of  combat  aircraft.  The  principal1 obstacle  to 
acceptance  of  unmanned  aircraft  is  uncertainty  about  their  costs. 


\X 


Escape  Systems 

In  some  air-to-air  and  air-to-ground  tactical  maneuvers,  depar¬ 
ture  from  controlled  flight  does  occur.  When  this  happens,  the  air¬ 
craft  may  be  at  a  speed  and  attitude  that  is  at  the  edge  of  the  safe 
ejection  envelope.  Even  when  ejection  is  possible,  given  the  speed 
and  attitude,  the  pilot  has  little  time  to  recover  the  aircraft  or 
to  eject  safely,  if  necessary. 

Air  Force  and  Navy  statistics  indicate  that  the  survival  rate  of 
pilots  who  ejected  from  combat  aircraft  declined  between  1976  and 
1980.  The  survival  rate  in  1976  was  85  percent,  in  1979  it  was  70 
percent,  and  in  1980  it  was  72  percent. 

In  addition,  during  1979  and  1980  there  were  as  many  ejections 
outside  the  escape  envelope  as  there  were  in  the  previous  three 
years.  The  fatality  rate  for  Air  Force  pilots  who  ejected  below  an 
altitude  of-  500  feet  was  57  percent  during  the  past  five  years. 

The  Advanced  Concept  Ejection  Seat  II  (ACES  II)  has  improved  the 
survival  rate  to  some  degree.  The  fatality  rate  is  still  unaccept¬ 
ably  high,  however,  and  solutions  to  this  problem  must  be  sought. 
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FINDINGS  AND  CONCLUSIONS* 


1.  The  complexity  of  today's  missions  and  high-performance  aircraft 
has  created  workloads  that  at  times  impose  intolerable  demands 
on  combat  pilots. 

2.  Air  Force  development  and  application  of  automated  features  can 
improve  operational  effectiveness  and  enhance  the  chances  for 
survival  of  pilots  and  combat  aircraft. 

3.  The  technology  for  automation  of  all  routine  tasks  and  of  some 
others  is  now  available.  Full  automation  is  costly  and  complex, 
however,  and  is  not  necessary  in  all  manned  combat  aircraft. 

A.  The  Air  Force  does  not  have  an  established  position  on  the 
requirements  for  automation  in  aircraft. 

5.  There  is  currently  no  systematic,  widely  applied  technology  for 
allocating  functions  between  automated  systems  and  the  pilot. 
Similarly,  there  is  no  criterion  for  balancing  the  costs  of 
automating  particular  functions  against  the  resulting  improve¬ 
ments  in  combat  performance. 

6.  Computer  technology  makes  it  possible  to  develop  dynamic,  inte¬ 
grated,  and  comprehensive  automated  systems  for  future  combat 
aircraft.  A  systems  approach,  emphasizing  the  core  function  of 
flight  trajectory  and  attitude  control,  is  a  logical  and  neces¬ 
sary  starting  point. 

7.  The  aircrews’  stated  immediate  need  is  for  improved  ability  to 
fly  low,  at  night,  and  during  severe  weather,  using  terrain  for 
cover  from  enemy  defenses.  The  critical  and  essential  functions 
that  could  be  automated  to  achieve  this  goal  have  not  been  com¬ 
pletely  identified,  although  current  programs  should  illuminate 
this  issue. 

8.  In  such  programs  as  AFTI-16  and  LANTIRN,  and  in  the  development 
of  technology  for  TF/TA,  the  Air  Force  research  and  development 
community  is  addressing  important  problems.  These  programs  will 
develop  technologies  and  an  engineering  perspective  that  are  a 
valuable  base  on  which  to  build.  The  approach  remains  piece¬ 
meal,  however,  and  without  clearly  stated  or  widely  understood 
objectives.  A  much-needed  unifying  focus  is  missing. 


•No  priority  ranking  is  intended  in  the  ordering  of  these  findings 
and  conclusions. 
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9. 


There  is  a  large  gap  between  what  Is  known  in  a  laboratory  set¬ 
ting  of  the  basic  characteristics  of  human  psychomotor  perfor¬ 
mance,  and  what  is  known  about  how  pilots  actually  fly  and  react 
in  modern  combat  aircraft.  Much  of  the  knowledge  needed  to 
design  an  automated  aircraft  that  uses  pilots'  skills  to  the 
best  advantage  lies  within  that  gap. 


10.  In  the  past,  the  unreliability  of  avionics  systems  has  been  a 
major  contributor  to  the  downtime  or  unavailability  of  combat 
aircraft.  No  effort  to  improve  combat  performance  by  further 
automation  can  succeed  without  adequate  attention  to  the  reli¬ 
ability  and  maintenance  of  the  equipment. 


11.  Fighter  aircraft  under  development  or  now  entering  the  inventory 
are  not  automated  to  the  extent  that  the  pilot  is  wholly  free  to 
assess  and  monitor  the  combat  situation  and  to  plan  his  further 
strategy.  No  aircraft  has  provided  him  with  effective,  acces¬ 
sible  aids  for  assessing  alternative  strategies. 

12.  Insufficient  attention  has  been  paid  to  past  efforts  at  automa¬ 
tion.  A  study  of  such  efforts  could  help  developers  to  repeat 
past  successes  and  avoid  past  shortcomings. 


13. 


Identification  of  unknown  objects  as  friend,  foe,  or  neutral 
(IFFN)  is  difficult  today.  IFFN  will  become  much  more  important 
in  the  future  because  of  improvements  in  weapons'  ranges. 


14. 


In  tactical  maneuvers  in  high-performance  aircraft,  pilots  often 
fly  at  the  edge  of  the  safe  ejection  envelope.  Current  auto¬ 
matic  ejection  equipment  is  inadequate  for  such  situations;  the 
number  of  injuries  and  fatalities  suffered  by  pilots  who  eject 
from  combat  aircraft  is  increasing. 
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RECOtOCATIONS 


1.  There  is  a  recognized  need  for  automation.  The  primary  goals 
should  be  to  increase  combat  effectiveness  to  enhance  survival 
of  pilots  and  aircraft,  and  to  decrease  pilot  work  load. 

2.  There  is  evidence  that  such  automation  can  be  available  in  the 
1990s.  A  firm  decision  can  and  should  be  made  to  automate 
specific  critical  functions  and/or  infrequently  performed  but 
essential  functions  that  are  currently  performed  manually. 

3.  A  systems-oriented  program  aimed  at  improving  and  developing 
automation  for  the  1990s  should  be  initiated  now.  The  goal 
should  be  a  core  design  that  would  form  the  basis  of  automated 
functions,  building  on  flight  trajectory  and  attitude  control 
systems.  Such  a  systems  approach  could  prevent  piecemeal  auto¬ 
mation  that  could  be  costly  and  would  result  in  only  partial 
solutions  not  adaptable  to  growth. 

4.  Four  functional  groups  are  promising  candidates  for  automation: 
(1)  flight  trajectory  and  attitude  control,  (2)  engine  and  power 
systems  control,  (3)  weapons  delivery  and  fire  control,  and  (4) 
navigation  and  communications  functions.  Combinations  of  these 
functional  families  can  be  accomodated  by  the  evolving  technol¬ 
ogy. 

5.  The  increasing  number  of  displays  used  to  present  information  to 
pilots,  the  amounts  of  information  and  instructions  displayed, 
the  limited  cockpit  area  available  for  display,  and  the  other¬ 
wise  complex  environment  of  the  aircraft  have  created  special 
problems.  Complicated  displays  are  difficult  to  read,  and  con¬ 
trols  and  functional  mode  selection  are  cumbersome  and  time- 
consuming.  Consequently,  necessary  actions  may  sometimes  be 
neglected.  To  reduce  pilot  workload  and  increase  operational 
effectiveness,  functions  that  divert  attention  from  critical 
actions  should  be  automated. 

6.  A  method  for  allocating  functions  between  automated  systems  and 
the  pilot  must  be  developed.  A  multidisciplinary  team  should 
examine  potential  hardware  and  software  techology,  as  well  as 
human  performance,  to  lay  the  basis  for  clear  decisions  in  this 
regard.  The  objective  should  be  a  practical  method  for  quanti¬ 
fying  the  improvements  in  performance  and  survival  that  result 
from  automating  particular  functions. 

7.  A  separate  and  fundamental  study  should  be  initiated  to  shed 
light  on  (1)  the  mental  model  pilots  create  to  aid  in  performing 
their  combat  tasks,  (2)  the  performance  characteristics  of  the 
controls  and  displays  through  which  the  pilot  und  automated 
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systems  interact,  and  (3)  human  capabilities.  This  study  should 
develop  a  multitask,  experimental  and  analytic  program  to  model 
pilot  behavior.  This  program  could  be  used  as  an  aid  in  design¬ 
ing  advanced  automated  systems,  and  in  particular  the  cockpits 
of  the  future. 

8.  Automating  or  partially  automating  a  higher  class  of  appropriate 
cognitive  functions,  such  as  the  ability  to  assess  the  combat 
situation,  or  to  plan  strategies  and  escape  routes,  should  be  a 
part  of  the  Air  Force's  long-range  program. 

9.  The  rising  trend  in  fatalities  and  serious  injuries  relating  to 
aircraft  escape  systems  indicates  a  need  for  improvements.  Air 
Force  activity  in  modifying  escape  systems  (ACJES  II)  may  meet 
this  need.  The  problem  must  be  addressed,  through  either  the 
ACES  II  program  or  a  completely  new  approach. 

10.  Identification  of  objects  for  beyond  visual  range  as  friend, 
foe,  or  neutral  (IFFN)  cannot  be  automated  with  any  confidence 
today.  An  automated  system  for  such  identification  would  permit 
important  gains  in  combat  effectiveness.  A  coordinated  effort 
on  this  front  is  needed. 
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ATTACHMENT  E.3 


CORRESPONDENCE 


DR.  JOSEPH  BIRT 


4711B  Shaunee  Creek  Dr. 
Dayton,  OH  45415 
October  30,  1985 


Mr.  Cyrus  Crites 
Chairman,  Executive  Board 
DOD  Human  Factors  Engineering 
Technical  Advisory  Group 
6520  Test  Group/ENAH  stop  239 
Edwards  AFB ,  CA  93523 

Dear  Cy  and  Members  of  the  Executive  Board, 

I  deeply  appreciate  the  acknowledgement  plaque  dated  May  8th 
which  the  Executive  Board  presented  to  me.  Last  minute  work 
demands  prevented  my  attendance  at  the  surprise  presentation 
meeting.  Thank  you  for  the  professional  recognition  and 
acknowledgement  by  many  friends  and  associates  who  signed  the 
plaque . 

The  TAG  has  some  unique  things  going  for  it.  These 
characteristics  are  the  secrets  of  its  success: 

-  grass  roots  motivation 

-  an  orientation  to  ongoing  and  planned  activities 

-  an  open,  management-free  environment 

You  need  to  protect  them  and  keep  up  the  good  work. 

Of  all  the  mememtos  I  received  upon  retiring  from  the 
military,  the  DOD  HFE  TAG  plaque  was  most  meaningful  to  me.  Thank 
you  for  your  kind  thoughtfulness.  As  you  may  know,  I  am  now  with 
the  Battelle  Columbus  Division  though  still  in  Dayton.  Although 
my  perspective  is  changing,  my  government  experience  was  indeed 
rewarding.  I  look  forward  to  future  affiliations  with  the  TAG  and 
with  my  military  and  civil  service  friends. 

With  high  regards  and  sincere  thanks,  I  remain 

Respectfully  yours, 

Birt 


cc:  Louida  D.  Murray 
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ATTACHMENT  F.l 


CONTROLS  AND  DISPLAY  —  MINUTES 


MINUTES  OF  THE  CONTROLS  AND  DISPLAYS  SUBTAG 


Submitted  by 
3.  D.  GROSSMAN 


The  SUBTAG  for  Controls  and  Displays  met  from  1830-2000,  on  5  November  1985  in 
San  Diego,  CA,  in  conjunction  with  the  main  TAG.  Approximately  35  people  were  in 
attendance. 

Three  presentations  were  made.  Dr.  Heather  Wild  described  a  study  she  has  designed, 
and  will  conduct  in  the  next  few  months  at  NADC,  on  CRT  display  size  requirements  for 
the  A-6F  aircraft. 

The  study  will  compare  the  proposed  A-6F  5x5-inch  display  versus  two  larger  sizes 
using  simulated  radar  imagery  on  the  display.  Dr.  Wild  will  evaluate  the  affect  of  size  on 
aircraft  safety  and  mission  effectiveness.  It  is  anticipated  that  Dr.  Wild  will  present  the 
results  of  this  study  at  a  future  meeting. 

Mr.  Steven  Kinsley  briefed  a  recent  NADC  study  on  F-18  HUD  pitch  ladder  formats. 
He  compared  performance  with  the  current  pitch  ladder,  against  a  proposed  replacement 
and  an  ADI  ball.  The  results,  given  the  test  conditions,  were  that  where  significant 
differences  in  performance  occurred,  performance  with  the  ADI  ball  was  best,  followed 
closely  by  the  current  format.  A  discussion  followed  concerning  the  ramifications  of  the 
test  design  on  the  results. 

Finally,  a  videotape  was  presented  by  Cy  Crites,  showing  a  liquid  crystal  flat  panel 
display  of  aircraft  engine  instruments  developed  by  GEC  Avionics. 

Following  the  presentation,  there  was  a  discussion  about  new  tasks  to  be  undertaken 
by  the  SUBTAG  in  recognition  of  the  ostensible  purpose  of  the  group,  which  is  to  provide 
technical  advice  on  controls  and  displays  to  the  DOD/NASA.  One  task  that  was  proposed 
would  be  to  identify  shortfalls  in  human  factors  guidelines,  data  and  technology  related  to 
controls  and  displays.  The  SUBTAG  expressed  a  clear  concern  that  we  are  currently  not 
in  a  position  to  provide  guidance  on  the  design  of  advanced  controls  and  displays  in 
aircraft,  spacecraft,  automated  land  vehicles  or  surface  and  subsurface  ships  now  on  the 
drawing  boards.  For  example,  there  is  a  highly  unsatisfactory  technology  base  relating  to 
controls  and  displays  for  multiple  sensors,  intelligent  systems,  and  systems  used  in  a 
weightless  environment. 

The  outcome  of  the  discussion  was  the  formation  of  a  working  group  that  will 
attempt  to  identify  these  control/displays  issues  of  the  future. 

One  meeting  of  the  working  group  has  been  held  and  another  is  scheduled.  The 
objective  of  the  group  will  be  to  develop  a  position  paper  on  both  near  and  long  term 
issues,  problems,  shortfalls  and  needed  technologies  for  advanced  controls  and  displays. 
The  position  paper  will  be  brought  to  the  SUBTAG  for  a  thorough  review  and  approval  and 
forwarded  to  the  main  TAG.  We  forsee  that  this  effort  will  1)  provide  the  basis  for 
directing  future  DOD/NASA  efforts,  2)  alert  DOD  and  NASA  management  about  TAG 
concerns  for  future  system  performance  that  depends  on  controls  and  displays,  3)  and 
provide  a  basis  for  future  SUBTAG  discussions  and  presentations. 

The  next  meeting  of  the  SUBTAG  will  take  place  in  conjunction  with  the  main  TAG 
meeting  in  May  '86.  It  is  proposed  that  the  meeting  be  held  on  Monday  of  the  week  of  the 
TAG  meeting.  Details  of  this  proposal  remain  to  be  worked  out. 
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ATTACHMENT  F.2 


A  COMPARISON  OF  F/A-18  HUD  AND  ADI  BALL  DISPLAYS 


ON  RECOVERY  FROM  UNUSUAL  ATTITUDES 


A  COMPARISON  OF  F/A-18  HUD 


AND  ADI  BALL  DISPLAYS  ON  RECOVERY  FROM 

UNUSUAL  ATTITUDES 


MR.  STEVEN  A.  KINSLEY 
MAN-MACHINE  INTEGRATION  DIVISION 
TECHNOLOGY  DEVELOPMENT  BRANCH 
NAVAL  AIR  DEVELOPMENT  CENTER 


PROBLEM 


F/A-18  MISHAPS: 

-  FLYING  INTO  WATER  ON  GCA  DOWNWIND  AT  NIGHT 

-  GROUND  IMPACT  DURING  ACM 

-  FLYING  INTO  GROUND  DURING  NIGHT  FCLP  WORK  UPS 
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LOSS  OF  SITUATIONAL  AWARENESS 


•  DURING  TACTICAL  MANEUVERING 

-  ACM 

-  AW  I 

-  BOMBING  (TARGET  FIXATION) 


•  DURING  NON-TACT  I  CAL  MANEUVERING  (I.E.,  CURRENT  GPWS) 

-  TAKEOFF/CATAPULT 

-  INSTRUMENT  APPROACH 

-  sssc 


•  AURAL  CUES 


-  RAD  ALT 

-  ENGINE 

-  ECS 


•  PITCH  ATTITUDE 


•  SEAT  OF  PANTS 


PRIMARY  ATTITUDE 
REFERENCE  (HUD) 


PILOT'S 

SITUATIONAL 

AWARENESS 


•  DISPLAYS 


-  HI  MAP 

-  RADAR 

-  FLIR 


•  FCS 


•  EXTERNAL  FIELD 
OF  VIEW 


FORCE  FEEDBACK 
NEUTRAL  SPEED  STABILITY 
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•  INS 


-  ATTITUDES 

-  NAVIGATION 


ADC 
-  BARD 


F/A-18  AIRCRAFT 
SITUATIONAL 
AWARENESS 


•  RADAR 

-  AGR  (DIRECTLY 
OR  SI DELOBES) 


-  G  AVAILABLE 

-  RATES  AVAILABLE 


F/A-18  PILOT’S  SITUATIONAL  AWARENESS 


•  FCS  -  MINIMAL  FEEDBACK  IN  AUTO 

•  PITCH  ATTITUDE  -  DOESN'T  CHANGE  WITH  AIRSPEED  CHANGES  (FLAPS  PROGRAM) 

•  HUD  -  NOT  ADEQUATE  FOR  QUICK  ATTITUDE  RECOGNITION 

•  AURAL  CUES  -  RAD  ALT  -  OK 

-  ENGINES/ECS  -  MINIMAL 

•  DISPLAYS  -  RADAR  -  EXP  DISPLAYS  DON'T  CONTRIBUTE  TO 

SITUATIONAL  AWARENESS 

-  FLIR  -  CAN  BE  VERY  DISORIENTING 

-  HI  (MAP)  -  GOOD  FOR  NAVIGATIONAL  SITUATIONAL 
AWARENESS 
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HUD  CUNltMS 


F/A-18  HUD 


PROPOSED  HUD 


5[ 


o 


15 

- 1 

,15 


ADI  BALL 


EXPERIMENTAL  PARADIGM 
(STATIC  EXPERIMENT) 


FORMAT 

F/A-18  HUD  ADI  BALL  PROPOSED  HUD 

PITCH 

PITCH 

PITCH 

ROLL 

0* 

60* 

120* 

180* 

-120* 

-60* 

0*  55*  -55* 

0*  55*  -55* 

O’  55*  -55* 

S1,S2,S3...S12 

S1,S2,S3. . .S12 

S1/S2/S3. , ,S12 
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MEAN  DECISION  TIME  AND  PERCENTAGE  ERRORS 
AS  A  FUNCTION  OF  DISPLAY  TYPE 


DEPENDENT 

MEASURES 

DISPLAY  TYPE 

F/A-18  HUD 

ADI  BALL 

PROPOSED  HUD 

MEAN  S.D. 

MEAN  S.D. 

MEAN  S.D. 

DECISION  TIME 

Ill  1.03 

3.37*  0.9M 

tl.W  0.99 

Z  ERRORS 

0.06  0.06 

0.09  0.11 

0.10  0.09 

*  SIGNIFICANTLY  DIFFERENT  (P  .001) 
FROM  OTHER  TWO  FORMATS 
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EXPERIMENTAL  PARADIGM 
(DYNAMIC  EXPERIMENT) 


ROLL 

O* 


60* 

120* 

180* 

-120’ 

-60* 


FORMATS 


F/A-18  HUD 

ADI 

BALL 

PROPOSED  HUD 

BACKGROUND 

BACKGROUND 

BACKGROUND 

DAY 

NIGHT 

DAY 

NIGHT 

DAY 

NIGHT 

PITCH 

PITCH 

PITCH 

PITCH 

PITCH 

PITCH 

O’  55*  -55* 

O’  55*  -55’ 

O’  55*  -55* 

O’  55*  -55* 

O’  55’  -55’ 

O’  55’  -55’ 

S 1-S12 

S1-S12 

S1-S12 

S1-S12 

S1-S12 

S1-S12 

MEAN  DECISION  TIME 


■j 
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MEAN  RECOVERY  TIME 


SBMB 


MEAN  RECOVERY  TIME  0 
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NUMBER  OF  ERRORS  AS  A 
FUNCTION  OF  DISPLAY  FORMAT 
AND  BACKGROUND  CONDITION 


DAY 


NIGHT 


F/A-18  HUD 

ADI  BALL 

PROPOSED  HUD 

ERRORS  -  23 

ERRORS  -  23 

ERRORS  -  19 

MEAN  -  2.091 

MEAN  -  2.091 

MEAN  -  1.727 

S.D.  -  2.071 

S.D.  -  1.640 

S.D.  -  1.272 

ERRORS  -  32 

ERRORS  -  36 

ERRORS  -  33 

MEAN  -  2.919 

MEAN  -  3.364 

MEAN  -  3.000 

S.D.  -  2.427 

S.D.  -  2.461 

S.D.  -  3.109 

65 


101 


TOTAL  -  55  TOTAL  -  59  TOTAL  -  52 
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ATTACHMENT  F.3 


EXPERIMENTAL  EVALUATION  OF  DISPLAY  SIZE 

FOR  THE  A-6F  AIRCRAFT 


EXPERIMENTAL  EVALUATION  OF 
DISPLAY  SIZE  FOR  THE 
A-6F  AIRCRAFT 


DR.  HEATHER  M.  WILD 
MAN-MACHINE  INTEGRATION  DIVISION 
TECHNOLOGY  DEVELOPMENT  BRANCH 
NAVAL  AIR  DEVELOPMENT  CENTER 
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PROBLEM 


•  CURRENTLY  5"  X  5"  CRT  DISPLAYS  ARE  PROPOSED 
FOR  THE  A-6F  AIRCRAFT 


•  NO  QUANTITATIVE  ASSESSMENT  OF  THE  REQUIRED 
A-6F  DISPLAY  SIZE 


OBJECTIVE 


•  CONDUCT  A  QUANTITATIVE  EXPERIMENT  TO  EXAMINE 
THE  EFFECT  OF  THREE  DISPLAY  SIZES  -  5"  X  5", 

6"  X  6",  7"  X  7"  -  ON  A-6F  OPERATOR  TARGET 
DETECTION  PERFORMANCE 


F.3-2 


BACKGROUND  (naDc) 

\s sssssav 


•  5"  X  5"  DISPLAYS  WERE  INTRODUCED  INTO  THE  A-6F 
AIRCRAFT  IN  THE  ABSENCE  OF  EXPERIMENTAL 
EVALUATION.  THE  PRIMARY  REASON  FOR 
IMPLEMENTING  THE  5"  X  5"  DISPLAY  IN  THE  A-6F 
WAS  TO  ESTABLISH  DISPLAY  SIZE  UNIFORMITY  WITH 
THE  F-14  AIRCRAFT 


•  SUBJECTIVE  REPORTS  FROM  PILOTS  AND  B/Ns  REVEAL 
THAT  A  LARGER  THAN  5"  X  5"  DISPLAY  IS  PREFERRED 


•  RESEARCH  HAS  DEMONSTRATED  A  GREATER  NUMBER 
OF  CRASHES  DURING  SIMULATED  HELICOPTER  FLIGHTS 
USING  13-cm  (5.12  in)  DISPLAYS;  HOWEVER,  A 
DECREASE  IN  THE  NUMBER  OF  CRASHES  WAS  NOTED 
AS  DISPLAY  SIZE  INCREASED  TO  26-cm  (10.24  in) 


•  RESEARCH  HAS  DEMONSTRATED  A  GREATER 

PROBABILITY  OF  TARGET  DETECTION  AS  DISPLAY 
SIZE  INCREASED  FROM  13-cm  (5.12  in)  TO  26-cm 
(10.24  in) 


•  RESEARCH  HAS  DEMONSTRATED  THAT  LOWER 
LUMINANCE  LEVELS  ARE  REQUIRED  WHEN  USING 
26-cm  (10.24  in)  DISPLAYS  THAN  WHEN  USING 
13-cm  (5.12  in)  DISPLAYS 


•  CONTRADICTORY  RESULTS  HAVE  DEMONSTRATED 
THAT  DISPLAY  SIZE  DOES  NOT  IMPACT  ON 
PERFORMANCE 


A-6  UPGRADE 
DISPLAYS 
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PILOT  MFD  #1  _ 

TERRAIN  FOLLOWING  AND  AVOIDANCE,  I  Umdc 
ENEMY  TERRITORY  - 


PRIMARY  DISPLAY 


EXPERIMENTAL  PARADIGM 

']jgggggggf 


DISPLAY  SIZE 

3"  X  3" 

6"  X  6" 

7"  X  7” 

ENVIRONMENTAL  CONDITION 

DAY 

NIGHT 

DAY 

NIGHT 

DAY 

NIGHT 

Sl 

Sl 

Sl 

Sl 

Sl 

Sl 

*2 

S2 

s2 

s2 

S2 

S2 

SUBJECTS 

• 

• 

, 

sn 

Sn 

s„ 

sn 

Sn 

•  DEPENDENT  VARIABLES 

-  NUMBER  OF  CRASHES  LOGGED  DURING  EACH  TRIAL 

-  MEAN  LUMINANCE  LEVEL  SET  DURING  FLIGHT 

-  PERCENTAGE  OF  TARGET  DETECTIONS 

-  REACTION  TIME  TO  TARGET  DETECTION 

-  MEAN  ALTITUDE  FLOWN  DURING  EACH  TRIAL 

•  FIXER  PARAMETERS 

-  AIR  SPEED 

-  DISTANCE  FROM  DISPLAY  TO  EYE  REFERENCE  POINT 

-  S/N  RATIO 

-  SYSTEM  GAMMA  FUNCTION 
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EXPERIMENTAL  PROCEDURE 


•  TRAINING  PHASE 

-  INSTRUCTIONAL/ FAMILARIZATION  BRIEFING 

-  SIMULATED  FLIGHT  PRACTICE 

it* 

W  •  TESTING  PHASE 

-  5  MIN  TRIALS 

-  MULTI-TASKING— ►  CONTINUOUS  FLYING  AND 
TARGET  DETECTION 

•  QUESTIONNAIRE 

-  WORKLOAD 


HYPOTHESES  PROPOSED  .(nm) 


•  THE  NUMBER  OF  CRASHES  OCCURRING  DURING 
SIMULATED  FLIGHT  WILL  DECREASE  AS  DISPLAY 
SIZE  INCREASES 


•  REACTION  TIME  TO  TARGET  DETECTION  WILL 
DECREASE  AS  DISPLAY  SIZE  INCREASES 


•  THE  PERCENTAGE  OF  TARGET  DETECTIONS  WILL 
INCREASE  AS  DISPLAY  SIZE  INCREASES 


•  THE  LUMINANCE  LEVEL  REQUIRED  FOR  VIEWING 
THE  DISPLAY,  PARTICULARLY  DURING  NIGHT 
CONDITIONS,  WILL  DECREASE  AS  DISPLAY  SIZE 
INCREASES 
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SIGNIFICANCE  OF  EXPERIMENTAL 

RESULTS 


•  OBJECTIVE  PERFORMANCE  DATA  FOR  SELECTION  OF 
DISPLAY  SIZE  IN  THE  A-6F 

-  BENEFIT  GRUMMAN  AND  NAVY  (A-6  PMA) 

•  THE  COMPATIBILITY  OF  THE  6"  X  6"  DISPLAY  WITH 
THE  5"  X  5"  DISPLAY  IN  USE  IN  THE  F-14  AIRCRAFT, 
IN  TERMS  OF  PARTS  AND  COST,  CAN  BE 
DEMONSTRATED 


POSSIBLE 

-  INCREASE 

-  DECREASE 

-  DECREASE 

-  DECREASE 


TARGETS  DETECTED 
TARGET  DETECTION  TIME 
DISPLAY  LUMINANCE 
AIRCRAFT  CRASHES 
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ATTACHMENT  G 


HUMAN  FACTORS  IN  LOGISTICS  —  MINUTES, 


ATTENDEE  LIST,  RELATED  MATERIALS 


MINUTES  FROM  THE  LOGISTICS  SUBTAG  GROUP 
MEETING  5-7  NOVEMBER  1985,  SAN  DIEGO,  CA 
CHAIR  MR.  DALE  MAHAR,  PACIFIC  MISSILE  TEST  CENTER 


1.  The  Logistics  SubTAG  met  late  Tuesday  night  and  was  attended 
by  a  small  group,  five  people  to  be  exact,  two  from  the  USAF,  one 
from  the  USA,  one  contractor  from  McDonnell  Douglas  Corporation 
and  myself  from  the  USN. 

2.  There  were  two  presentations  given;  they  both  were  very  good 
and  informative.  The  first  was  by  Mr.  Sheldon  Levine  of  McDonnell 
Douglas  Corporation  on  Design  for  Maintainability  and  the  second 
by  Ms.  Wendy  Campbell  of  Wright-Patterson  AFB  on  the  Lessons 
Learned  Program. 

3.  Following  the  presentations,  we  entered  into  a  discussion  on 
whether  the  Logistics  SubTAG  should  be  cancelled  due  to  the 
general  lack  of  interest.  After  much  discussion,  it  was  decided 
that  a  name  change  appeared  to  be  the  correct  direction  in  which 
to  proceed.  After  several  suggested  names,  it  was  agreed  by  all 
that  were  present,  that  the  new  proposed  name  would  be  Design  for 
Supportability . 

4.  The  Design  for  Supportability  name  was  agreed  upon  due  to  the 
fact  that  it  was  more  general  in  nature.  It  was  generally  agreed 
that  Logistics,  Design  for  Maintainability,  Supportability,  HFE, 
and  training  should  be  more  integrated. 

5.  If  there  are  any  objections,  changes  or  suggestions  from  any 
member  of  the  Logistics  SubTAG  group  that  were  not  able  to  attend 
this  meeting  or  from  members  of  the  mother  TAG,  please  get  in 
touch  with  me.  Otherwise,  I  will  initiate  the  agreed  upon 
changes . 

6.  Enclosures  (1)  LOGSTAG  Agenda,  (2)  Attendance  List,  and  (3) 
Mr.  S.  Levine's  viewgraphs  from  his  presentation. 

7.  I  wish  to  thank  those  that  attended  the  Logistics  SubTAG  and 
especially  Ms.  Wendy  Campbell  and  Mr.  Sheldon  Levine  for  their 
presentations. 


DALE  MAHAR 

Logistics  SubTAG  Chair 
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LOGSTAG  AGENDA 
5  November  1985 


1830-2000  Teak  Room 


Opening  Remarks:  Dale  Mahar,  LOGSTAG  Chair 
Presentations ; 

1.  Mr.  Sheldon  Levine,  McDonnell  Douglas  Corporation,  St. 
Louis,  MO 

Topic:  Design  for  Maintainers 

2.  Ms.  Wendy  Campbell,  Wright-Patterson  AFB,  OH 

Topic:  Lessons  Learned  Program 

Group  Questions/Discussion 


Old  Business 


New  Business 


Direction(s)  LOGSTAG  Should  Proceed 


Action  Items 


Closing  Remarks 
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Many  of  today's  military  systems  cannot  be  adequately 

OPERATED,  MAINTAINED,  OR  SUPPORTED  BECAUSE  THE  DEPARTMENT 

of  Defense  does  not  pay  enough  attention  to  logistic 

SUPPORT,  HUMAN  FACTORS,  AND  QUALITY  ASSURANCE  DURING 
THE  DESIGN  PHASE  OF  THE  ACQUISITION  PROCESS. 


Designers  of  weapon  systems  often  do  not  consider  the 

CAPABILITIES  AND  PERFORMANCE  REQUIREMENTS  OF  THE  PEOPLE 

WHO  WILL  OPERATE  AND  MAINTAIN  THE  SYSTEMS  IN  THE  OPFRAT IONAI 

ENVIRONMENT.  It  IS,  THEREFORE,  EXTREMELY  IMPORTANT 

THAT  WEAPON  SYSTEM  REVIEWS  INCLUDE  AN  EVALUATION  OF  THE  EXTENT 

TO  WHICH  DOD  CONSIDERS  MANPOWER,  PERSONNEL,  AND  TRAINING 

NEEDS  AND  CAPABILITIES  IN  THE  WEAPON  SYSTEM  ACQUISITION 

PROCESS. 


HUMAN- INDUCED  MALFUNCTIONS  MAY  EE  A 

VERY  SERIOUS  PRQBLEa 

The  HUMAN  LIMITATIONS  OF  THE  OPERATOR  AND  MAInTAINER  CAN 
CAUSE  SYSTEM  FAILURES.  The  EXTENT  OF  THIS  PROBLEM,  HOWEVER, 
CANNOT  BE  ACCURATELY  ESTIMATED  FROM  DATA  IN  CURRENT  FAILURE 
REPORTING  SYSTEMS.  BASED  ON  INTERVIEWS  WE  CONDUCTED  AND 
STUDIES  WE  REVIEWED,  WE  ARE  CONVINCED  THAT  A  LARGE  NUMBER 
OF  SYSTEM  FAILURES  ARE  THE  RESULT  OF  HUMAN  ERROR.  We  FOUND 
A  DOD  AND  CONTRACTOR  STUDY  WHICH  SHARE  THIS  POSITION.  BOTH 
ESTIMATE  THAT  HUMAN  ERRORS  ACCOUNT  FOR  AT  LEAST  50  PERCENT 
OF  THE  FAILURES  OF  MAJOR  SYSTEMS. 


G-3 


The  increasingly  complicated  nature  of  modern  military 

SYSTEM  TOGETHER  WITH  INTERNAL  MILITARY  PERSONNEL  PROBLEMS 
SUGGESTS  THAT  HUMAN-INDUCED  ERRORS  BOTH  IN  OPERATIONS 
AND  MAINTENANCE  COULD  ALSO  INCREASE  UNLESS  MORE  ATTENTION 
IS  PAID  TO  THIS  PROBLEM  DURING  DESIGN  AND  DEVELOPMENT, 


GAO  Conclusion  -  "If  the  design  of  systems  is  to 

ADEQUATELY  CONSIDER  ALL  THE  HUMAN  LIMITATIONS  (INCLUDING 
SKILL  LEVELS,  PROFICIENCY,  AVAILABILITY,  ENVIRONMENTAL 
STRESS,  AND  FATIGUE),  MILITARY  SPECIFICATIONS,  STANDARDS, 
AND  HANDBOOKS  MUST  ADDRESS  THESE  FACTORS.  EXISTING 
DOCUMENTS  DO  NOT.* 
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DEPARTMENT  OF  DEFENSE 
HUMAN  FACTORS  ENGINEERING 
TECHNICAL  ADVISORY  GROUP 
5-7  NOVEMBER  1985 
LOGSTAG  ATTENDANCE  LIST 
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Design  for  Maintainer 
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SHELDON  H.  LEVINE 
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HUMAN  ENGINEERING  DESIGN 
APPROACH  DOCUMENT-MAINTAINER 

(HEDAD-.M) 


ANALYSIS  WORKSHEETS 

EQUIPMENT  DESCRIPTION 
DESIGN  FOR  MAINTA1NER 

FOOLPROOFING 
TASK  PROCEDURES 


EQUIPMENT  DESCRIPTION 


EQUIPMENT  DATA 
FUNCTION 

MAINTENANCE  TASKS 
PHYSICAL  CHARACTERISTICS 

MOUNTING  PROVISIONS 
CONNECTIONS 


EQUIPMENT  ENVELOPE 


u/r,A.  LEFT  PITOT  AND  STATIC  2  MOISTUKZ  TRAPS 


ohi-oooj-jo* 
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WAA; 


LEFT  mOT  AND  STATIC  2  MOISTURE  TRAPS 


VISUAL  ACCESS;  Visual  access  is  ehrough  door  122L,  39  inches  off  Che  deck 
Drains  ere  located  3  inches  below  che  door  sill  and  12  inches  inCerior  to 
Che  door  openiog.  Drain  parta  are  acccsaed  froa  the  ao Id  line  34  ioches 
above  che  deck.  Visual  access  will  be  obscured  by  che  hands  when  reaching 
Chrough  Che  door  opening.  Drain  is  replaced  froa  aoldlioc  aod  has  direct 
visual  access  froa  outside  che  aircrafe. 


PHYSICAL  ACCESS:  Physical  access  is  through  door  122L,  an  6  by  14  inch 
opening  39  ioches  above  the  deck.  A  fifth  percentile  aaintainer  will  need 
a  vorkataod  Co  reach  che  water  trapa  which  ere  12  inches  ioterior  and  3 
inches  belov  the  door  sill.  Moisture  Crap  drain  valves  are  accessed 
direedy  froa  the  oucside  eoldline  end  ere  io  reach  of  Che  fifth  through 
ninecy-fifch  percencile  aaintainer  population. 


LIFTING/CAAAYING:  Swell  end  light  enough  to  be  carried  by  one  man. 


SAFETY:  Bo  additional  safety  features  required. 


MAINTENANCE  CONTAOLS/OISPLAYS:  Pores  on  Craps  ere  labeled  IN  end  OUT. 


FOOLPROOFING 

REMOVAL 

INSTALLATION 

VERIFICATION 
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FOOLPROOFING 


WRA: 


LEFT  PITOT  AND  STATIC  2  MOISTURE  TRAPS 


FAILURE  INDICATION:  Cracked  or  damaged  Craps  are  detected  by 
visual  inspection.  Pressure  check  of  pitot  static  system 
will  also  isolate  faulty  traps.  Drain  valve  failure  will  be 
indicated  by  an  accumulation  of  water  in  the  traps  and  no 
flow  when  traps  are  emptied  during  daily  inspection. 


REMOVAL* 

Drains  are  removed  in  direct  line  from  mounting 
bracket.  Traps  are  of  convenient  size  for  one  hand  removal. 
Tubes  are  removed  before  mounting  bolts.  Flats  are  provided 
on  adapter  to  hold  trap  steady  while  loosening  connector. 

To  disconnect  tube  from  moisture  trap,  loosen  connector  nut 
and  pull  tubing  free.  Removing  connector  nut  can  result  in 
loose  parts  internal  to  the  fitting  being  lost. 


REPLACEMENT:  Connectors  are  sized  so  as  not  to  cross  connect 
pitot  and  static  systems.  Drain  valve  opening  in  mounting 
bracket  ensures  proper  orientation  of  trap.  Bolts  and  nuts 
are  installed  in  either  orientation.  Procedure  for  tightening 
connector  fittings  is  outlined  in  Technical  Manuals  Adapter 
and  connectors  both  must  be  held  steady  while  tightening. 


INSTALLATION  VERIFICATION:  Visually  inspect  traps.  Pitot 
static  pressure  test  to  ensure  integrity  of  the  system. 


OMI-OOtt'K* 
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REMOVAL /INSTALLATION  PROCEDURE 


WRA-  LEFT  t*ITOT  AND  STATIC  2  MOISSTUBX  TRAPS 


PERSONNEL 
NUMBER:  i 


NEC/M  OS:  AE83XX/6335 


SKILL  LEVEL:  B 


SUPPORT  EOUIPMENT/MATERIALS:  miL-G-4343  Aircraft  pm,  MS28775-113 

packing  for  pitot  static  trap,  torqua  wrench  (5  to  30  lbs), 
wrenches,  MIL-S-8660  silicon*  compound  ,  MS28775-6  packing 
for  drain  valve. 


1.  REMOVAL: 

1.1  0p«n  door  122L  (28  Hi  Torqu*  fastcnars). 

1.2  Loosco  connector  (2)  fittings  (about  1/2  turn)  and  pull 
tub*  fr«*  of  fitting. 

1.3  Remove  outs  and  remove  bolts  from  mounting  adapter. 

1.6  Remove  trap. 

1.3  Remove  connector  fittiogs  from  trap. 

1.6  Remove  old  packing  from  connector  fittings. 


Note:  To  remove  drain  valve  unscrew  using  3/8  io  socket 
wrench. 


2.  INSTALLATION: 

2.1  Lubricate  new  packing  and  iostall  on  adapters. 

2*2  Install  coooector  fittiogs  into  trap  and  tighten 
1/4  turn  past  fioger  tight. 

2.3  Install  trsp  into  mounting  adapter  on  aircraft  (2  bolts 
washers  and  outs). 

2.4  Install  tubes  into  connector  fittiogs  and  tighten  per 
Tcchoical  Manual. 

2.3  Perform  pitot  static  pressure  test. 

2.6  Secure  door  122L  (28  Hi  Torque  fasteners). 

Note:  To  replace  drain  valve,  grease  packing,  install  packing 
oo  drain  valve,  apply  coating  of  silicone  compound  to 
threads,  iostall  into  trap  and  torque  to  20-30  inch-pounds. 


100%  COMPLETED  AND 

DELIVERED  PRIOR  TO 
FIRST  FLIGHT  < 
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FOOLPROOFING  -  COVERAGE 


FAILURE  IDENTIFICATION 


Failed  WRA  identified 

Pilot  Report 

Bit 

Inspection 

Warning /Caution/Advisory 

Scheduled  Change 

Isolate  failed  WRA  for  R/R 

Maintenance  Bit 

Troubleshooter  per  T.O. 

WRA  failure  Indicator 

Door /Shelf /Part  labels  to 
locate  part 

REMOVAL 


Mounting  Design 

FOD  potential,  dropped 
fasteners  and  lockvlre  etc. 

Blind  access  -  hidden  fasteners/ 
connectors 

4r  .  ' 

« 

Direct  line  removal-damage 
to  WRA  and  adjacent  objects 
Handling  of  WRA  during  removal 
Order  of  removal  of  connectors  and 
fasteners 

Lighting  and  carrying  placards 

Connectors 

Clearance  of  hard  lines 
during  removal 

Loose  hardware  and  parts 
Consistency  of  operation  for 
removal 

Force  requirements  for  removal 
Interlock  on  removal  from 
rack  or  tray 

Placard  on  safety 

Pressure  relief  (hydraulics) 

REPLACEMENT 


Mounting  Design 

Installation  orientation  and 
alignment 

Interchangeability 

FOD 

Installation  sequence 

Torque  Placards  and  Labeling 
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MTA  itch  ocsciirrioa 

2.  IDCNTiriCATlON  *>($>. 

1.  TIHC 

Human  Engineering  Design  Approach  Document-Malntalner 

D00 

DI-H-7057 

s.  KscnieTi«/n*Tou 

This  document  provides  a  source  of  data  to  evaluate  the 
extent  to  which  equipment  having  an  Interface  with  main¬ 
tained  meets  human  performance  requirements  and  human 
engineering  design  criteria. 

A.  ATMOVAt.  DAT! 

1  June  1979 

S.  orritt  or  mimast 
scsroNSiaiUTV 

ARMY /MIRAD  COM 

a.  ooc  scouitco 

a.  A^SOVAL  LIMITATION 

7.  APrwiCATION/INTIMn.ATIONSMlP 

This  DID  replaces  DI-H-2108  and  roi-H-21385. 

This  DID  is  primarily  applicable  to  vork  tasks  delineated 
in  pArA|i tph(i)  3. 2. 1.2,  3. 2. 1.3,  3. 2. 1.4,  ud  3.2.2  of 
K1L-H-A6835B. 

1.  •CFCHCNCCS  IMANOATOffr  AS  Cl  T£0  IN 
■.OCX  101 

MIL-H-46855B 

MIl-STO-1472 

MCSL  *J*a£»(S) 

10.  MPAAATItt*  INITWJCTIONS 


10.1  General .  The  Human  Engineering  Design  Approach  Document  -  Maintainer  (HEDAD- 
M)  shall  be  prepared  which  describes  the  characteristics,  layout,  and  Installation  of 
all  equipment  having  a  maintainer  Interface  (excluding  depot  level  maintenance  ac¬ 
tions);  It  shall  also  describe  maintainer  tasks  associated  with  the  equipment.  The 
HEDAD-M  shall  describe  the  extent  to  which  the  requirements  of  MIL-STD-1472  and  other 
applicable  human  engineering  documents  specified  In  the  contract  have  been  Incorporated 
Into  the  design,  layout,  and  Installation  of  equipment  having  a  maintainer  Interface. 
Maintainer  task  analysis  results  shall  be  presented  as  part  of  the  rationale  supporting 
the  layout,  design  and  Installation  of  the  equipment.  The  requirement  for  this  Infor¬ 
mation  Is  predicated  on  the  assumption  that,  as  analytic  and  study  Information,  It  Is 
developed  sufficiently  early  to  Influence  the  formulation  of  other  system  data  such  as 
maintenance  allocation  charts,  special  repair  parts/tool  lists,  LSAR  data.  If  the 
program  has  progressed  to  the  point  where  the  required  data  Is  available  through  other 
reporting  media,  such  as  those  noted  above,  they  shall  not  be  duplicated  but  shall  be 
referenced  or  appended  to  the  HEDAD-M  along  with  appropriate  supplementary  Information 
fulfilling  the  Intent  of  this  provision. 

10.2  Content  Requirements.  The  HEDAD-M  shall  consist  of  the  following  information: 

1)  List  of  each  Item  of  equipment  having  a  maintainer  Interface  at  the  Or¬ 
ganizational  and  FI eld/ Intermediate  Maintenance  Activity  (IMA)  level,  a  brief  statement 


oo ,  \m 


1  or  3  rtcii 
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DI-H-7057 

10.  PREPARATION  INSTRUCTIONS  (continued) 


task  number,  task  title,  task  frequency  (for  scheduled  maintenance 
actions)  or  estimated  task  frequency  (based  on  equipment  mean-tlme- 
between-fallure  for  unscheduled  maintenance  actions),  data  source  used 
(e.g.,  drawing  number,  sketch  number,  development  hardware,  actual 
production  equipment),  detailed  task  sequence  (see  paragraph  6.2.5  of 
MIL-H-46855B),  support  equipment  required,  tools  required,  job  aids 
required,  estimated  task  time,  estimated  personnel  requirements  (e.g., 
number  of  personnel  required,  skills  and  knowledge  required)  and  human 
engineering  considerations  which  reflect  specific  human  engineering 
requirements  Incorporated  Into  the  design  (e.g.,  malntalner  fatigue, 
potential  hazards,  safety  or  protective  clothing/equipment  required  or 
recommended,  access  problems,  malntalner  communication  requirements, 
special  task  sequence  requirements,  labeling).  As  applicable,  the 
following  types  of  malntalner  tasks  shall  be  addressed  by  task  analyses: 
remove/replace,  trouble-shoot  (fault  location),  repair,  adjust.  Inspect, 
service  and  test.  Critical  tasks  (see  paragraph  6.2.1  of  MIL-H-46855B) 
shall  be  clearly  Identified. 

7)  Narrative  which  provides  rationale  for  any  need  to  deviate 
from,  or  take  exception  to,  MIL-STD-1472  or  other  contractual  Item  human 
engineering  requirements. 

8)  Two  sketches,  drawings  or  photograph  of  each  of  equipment 
having  a  malntalner  Interface.  Each  Item  of  equipment  shall  be  depicted, 
a)  by  Itself  from  top,  front  and  side  (three- view  trl metric  or  exploded 
trimetric  view)  and  b)  Installed  as  the  malntalner  would  normally  view 

It  during  maintenance. 

9)  Sketches,  drawings  or  photograph  of  each  Item  of  equipment 
being  considered  as  alternatives  to  the  selected,  or  baseline  design. 
Sketches,  drawings  or  photographs  of  alternative  equipment  Installations 
or  layouts  which  exist  at  the  time  of  HEDAD-M  preparation. 

10)  Description  of  design.  Installation  or  layout  changes  which 
have  been  made  since  the  last  HEDAD-M  submission. 

10.3  Format  and  Data  Organization  Requirements.  The  HEDAD-M  be 
prepared  In  contractor  format  except  that  Information  shall  be  presented 
In  two  major  parts: 

1)  Information  pertaining  to  maintenance  actions  performed  at 
the  Organizational  Level. 

2)  Information  pertaining  to  maintenance  actions  performed  at 
the  Fleld/IMA  level. 


*ua.  oonnmt  nurnnc  omc* 
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ATTACHMENT  H 


HUMAN  FACTORS 

ATTENDEE 


TEST  AND  EVALUATION  —MINUTES 

LIST,  RELATED  MATERIALS 


Minutes  of  the  Test  and  Evaluation 
Sub-Technical  Advisory  Group  Meeting 
of  5  November  1985 

The  Test  and  Evaluation  Sub-Technical  Advisory  Group  (T&E  STAG)  met  on  5 
November  1985.  This  meeting  was  held  in  conjunction  with  the  15th  meeting  of 
the  Department  of  Defense  Human  Factors  Engineering  Technical  Advisory  Group 
(HEF  TAG)  in  San  Diego,  California.  The  meeting  of  the  T&E  STAG  was  limited 
to  three  hours,  but  much  was  accomplished. 

Approximately  50  people  attended  the  meeting,  42  of  which  signed  the 
attendance  sheet  (enclosure  (1)).  Keith  Karn  (Naval  Air  Test  Center)  chaired 
the  meeting  and  wrote  these  minutes,  filling  in  for  CDR  William  Moroney,  the 
T&E  STAG  Chairman,  who  was  unable  to  attend. 

Keith  opened  the  meeting,  welcomed  attendees,  presented  the  agenda  shown 
in  enclosure  (2),  announced  the  open  and  the  government  only  sessions,  and 
gave  a  brief  update  on  test  and  evaluation  equipment  and  procedures  under 
development  at  the  Naval  Air  Test  Center. 

Two  invited  presentations  were  made  by  people  from  Essex  Corporation. 
Dr.  Betina  Babbit  gave  a  presentation  on  the  construction  of  questionnaires. 
The  presentation  was  based  on  her  recently  published  questionnaire 
construction  manual.  The  presentation  and  the  manual  cover  topics  including 
types  and  content  of  questionnaires,  attitude  scales,  anchoring, 
administration  of  questionnaires,  characteristics  of  respondents,  and 
evaluating  results.  Her  presentation  is  enclosure  (3).  The  manual  has  been 
printed  by  Essex  Corporation.  It  was  written  in  conjuction  with  Charles 
Nystrom  of  the  Army  Research  Institute.  The  report  will  be  available  through 
DTIC  soon. 

Roger  McTighe  also  of  Essex  Corporation  presented  the  Essex  Training 
Analysis  System  (enclosure  (4)).  This  is  a  micro-computer  based  data 

collection,  manipulation,  and  reporting  system  designed  to  aid  the  human 
factors  engineer  in  developing  job  analyses,  task  analyses,  job  performance 
measures,  and  training  objectives.  Although  the  system  was  developed  for  the 
Nuclear  Power  Industry,  its  flexibility  was  demonstrated  by  using  it  to 
develop  a  task  analysis  for  a  tune  up  of  a  1979  Hona  Civic.  Demonstration 
Disketts  are  available  from  Essex.  The  system  runs  on  IBM  PC  and  compatible 
systems . 

After  these  two  Essex  presentations,  the  remaining  portion  of  the  meeting 
was  for  government  personnel  only,  at  the  request  of  the  T&E  STAG  chairman. 

Dr.  Dave  Meister  (Naval  Personnel  Research  and  Development  Center) 
presented  the  results  from  a  recent  survey  of  human  factors  professionals  in 
test  and  evaluation.  His  survey  covered  the  frequency  of  use,  utility,  and 
ease  of  use  of  various  test  and  evaluation  methods,  practices,  and  procedures. 
He  found  a  high  correlation  between  frequency  and  ease  of  use  and  also  a  high 
correlation  between  frequency  of  use  and  utility.  He  also  presented 
interesting  responses  on  constraints  that  practicing  human  factors  engineers 
face  in  the  test  and  evaluation  environment.  His  presentation  is  enclosure 
(5). 
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Ms,  Jan  Stoliker  (Edwards  Flight  Test  Center)  discussed  her  work  on  the 
AFTI  F-16.  She  is  currently  involved  in  obtaining  man-in-the-loop  system 
performance  data  from  the  1553  data  bus.  She  is  collecting  information  about 
target  tracking  accuracy  (comparing  automatic  vs,  manual  modes),  airspeed, 
altitude,  and  flight  control  position,  Jan  is  also  collecting  workload  data 
using  biomedical  measures  and  subjective  ratings.  Although  the  results  will 
not  be  available  until  a  future  meeting,  these  methods  appear  to  be  useful 
and  demonstrate  great  potential  for  tapping  into  the  1553  bus  for  a  wealth  of 
data  (see  enclosure  (6)). 

Dr.  Jim  Geddie  (Army  Human  Engineering  Lab)  gave  a  brief  update  on  the 
task  analysis  handbook  and  proposed  military  standard.  Drafts  of  these 
documents  are  in  review  by  an  AD  HOC  committee.  The  T&E  STAG  membership  may 
see  a  somewhat  smoother  version  by  the  next  meeting.  After  review  by  the  T&E 
STAG,  the  proposed  military  standard  will  be  sent  to  the  standardization 
committee . 

Mr.  Tom  Enderwick  (Naval  Personnel  Research  and  Development  Center)  made  a 
short  presentation  on  interaction  between  the  T&E  STAG  and  the  Test  and 
Evaluation  Technical  Group  of  the  Human  Factors  Society.  Tom  is  active  in 
both  groups.  He  presented  the  purpose  and  activities  of  the  Human  Factors 
Society  T&E  Group  and  invited  participation  (enclosure  (7)). 

The  meeting  ended  with  a  discussion  of  new  business.  Dr.  Jim  Geddie  was 
selected  as  the  new  chairman  of  the  T&E  STAG.  The  group  also  passed  a  motion 
to  adopt  the  concept  of  chair-select  as  many  of  the  other  SUB-TAGs  have.  This 
allows  for  an  individual  to  be  identified  for  the  chairmanship  a  year  in 
advance.  During  the  year  prior  to  his/her  chairmanship,  this  person  will  act 
as  an  assistant  to  the  incumbent  chairperson.  Selection  of  an  Air  Force 
representative  for  the  chair  select  position  should  occur  prior  to  the  next 
T&E  STAG  meeting. 

The  T&E  STAG  was  invited  to  the  Human  Engineering  Laboratory,  in  Aberdeen, 
Maryland  for  the  next  meeting.  This  meeting  will  be  held  independently  of  the 
main  TAG  meeting  in  April  1986. 

Overall,  this  was  a  productive  meeting.  The  attendance  list  shows 

participation  from  many  research  and  development  centers  as  well  as  the  test 
and  evaluation  groups.  This  type  of  participation  emphasizes  the  importance 
of  test  and  evaluation  during  all  phases  of  product  development. 

NOTE:  The  rationale  for  declaring  several  of  the  presentations  "Government 
Only"  was  questioned  during  the  meeting  of  the  operating  board.  It  was 
decided  that  for  future  discussions  the  criteria  for  the  "Government  Only" 
designation  should  be  the  same  as  the  criteria  used  for  classifying  reports. 
These  minutes  include  copies  of  viewgraphs  from  all  presentations  made  at  this 
meeting  (including  the  "Government  Only"  session.)  This  was  done  with  the 
permission  of  the  presenters  as  they  determined  that  no  contractor-sensitive 
data  was  included. 
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Human  Factors  Test  and  Evaluation  SubTAG 


Attendee  List 
5  November  1^85 


Mr  .  Homer  Adk i ns 
Code  N- 131 ;  Bldg.  654 
Naval  Air  Station 
Pensacola,  FL  32508 
<904)  452-3591 
922-3591 

Mr.  Richard  Armstrong 

P.0.  Box  716 

ATTN:  SLCHE-FR 

Ft.  Rucker,  AL  36362 

<205)  255-3303/2069 

558-3303 

Dr.  Bettina  Babbitt 
Essex  Corp  . 

Westlake  Village,  CA  91361 
<818)  889-5072 

Mr.  Donald  Bamber 

Navy  Personnel  R&D  Center 

Code  71 

San  Diego,  CA  92152 
<619)  225-2081 
933-2081 

Major  Tom  Bennett 
USA  CDEC 

ATEC-TP  ATTN:  T.  Bennett 
Ft.  Ord,  CA  93941 
<408)  242-6309 
929-6309 

Mr.  James  Broyles 
NPRDC 

5058  Ducos  Place 
San  Diego,  CA  92124 
<619)  225-6500 
933-6500 

Ms.  Wendy  Campbell 
AF  ALC/EREM 

Wr i gh t-Pat terson  AFB,  OH  45433 

<513)  255-4770 

785-4770 

Mr.  Gerald  Chaikin 

HEL  Det.-MICOM,  SLCHE-MI  <Chaikin) 
US  Army  Missile  Command 
Redstone  Arsenal ,  AL  35898-7290 
<205)  876-2048 
746-2048;  FT S  876-2048 


Ms.  Sherry  Chappell 
NASA -Ames 
NASA  239-21 

Moffett  Field,  CA  94035 
<415)  694-6909 
FTS  464-6909 

Mr  .  Cyrus  Cr i tes 
6520  TESTG-ENAH 
Stop  239 

Edwards  AFB,  CA  93523 
<805)  277-3334 

350- 3334 

Cap t .  Robert  L.  Dotson 
HQ  BMO/AWRH/Ch  i  e-f  ,  HFE  DIV 
Norton  AFB,  CA  92409 
<714)  382-7067 
876-7067 

Mr.  Thomas  P.  Enderwick 

Code  71 

NPRDC 

San  Diego,  CA  92152 
<619)  225-6617 
933-6617 

Dr.  Carl  E.  Englund 

Naval  Health  Research  Center 

P.0.  Box  85122 

San  Diego,  CA  92138-9174 

<619)  225-7393 

933-7394 

Dr.  James  C.  Geddie 
USAHEL  Field  Office  < T CATA ) 

HQ  T  CAT  A ,  ATTN:  SLCHE-FH< Gedd i e) 
Ft.  Hood,  TX  76544-5065 
<817)  288-9917/9921 
738-9921/9917,  FTS  788-9917 

Major  Lucky  Goebel 

B- 1 BFOT&E/TDD 

Dyess  AFB,  TX  79606-5000 

<915)  696-4433 

461-4433 

Mr .  Herb  Gol dman 
PMTC 

Code  4025 

Point  Mugu ,  CA  93042 
(605)  989-9538 

351- 9538 
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Mr .  Tom  Greene 

United  States  Army  Infantry  Board 
ATZB-I B-AE-HF  <T.  Greene) 

Ft.  Benning,  GA  31905 
(404)  545-3691/3621 
835-3691/3621 


Mr.  Judah  Katznelson 
( USA I B)  US  ARI  ,  ATTN:  PERI-SM 
5001  Eisenhower  Ave. 
Alexandria,  <JA  22333-5600 
(202)  274-8877 
284-8877 


Dr.  E.  M.  Haver  1  and 

USA  OTEA,  ATTN:  CSTE-ASD-AA 

5600  Columbia  Pike 

Falls  Church,  VA  22041-5115 

(202)756-2293/2294 

289-2293/2294 


Mr.  Steve  A.  Kinsley 

Naval  Air  Development  Center 

Code  6021 

Warminster,  PA  18974-5000 
(215)  441-7115 
441-7115 


Mr.  Ernest  Horton,  Jr. 

STEYP-MT-ES 

USA  Yuma  Proving  Ground,  Test  Eng. 
Yuma,  AZ  85365-9103 
(602)  328-3298 
899-3298,  FTS  765-3298 

LTC  Frederick  A.  Isgrig 

HQ  TECOM 

(AMSTE-EV-H) 

Aberdeen  Proving  Ground,  MD  21005“ 
(301)  278-5145/5597 
298-5145/5597 

Mr.  Ralph  L.  Jarboe 

USAARENBD 

ATTN:  ATZK-AE-TR 

Ft.  Knox,  KY  40121-5470 

(502)  624-5973 

464-6965,  FTS  354-6965 

Mr  .  Monty  Joe  1 
ESD/ALET 

Hanscom  AFB ,  MA  01731 
(617)  861-4732 
478-4732 

Mr.  Todd  Jones 

USCG ,  Office  o-f  R&D ,  G-DMT/-1 
2100  Second  St.,  S.W. 

Washington,  DC  20593 
(202)  426-1058 
FTS  426-1058 


COL  James  Leach 
D0T&E ,  OSD,  Room  1A  1063 
Div.  The  Pentagon 

Washington,  DC  20301-1700 

(202)  697-3891 

227-3891 

Mr.  Rodger  McTighe 
Essex  Corp . 

3211  Jefferson  St. 

5055  San  Diego,  CA  92110* 

(619)  299-8043 

Dr.  David  Meister 

Code  71 

NPRDC 

San  Diego,  CA  92152 
(619)  225-6617 
933-6617 

Mr.  John  L.  Miles,  Jr. 

ARI,  ATTN:  PERI-SM 
5001  Eisenhower  Ave . 
Alexandria,  UA  22333-5600 
(703)  274-8917 
284-8917 

Mr.  Douglas  Miller 
HQ  SD/ALT  (USAF) 

P.0.  Box  92960  WPC 

Los  Angeles,  CA  90009-2960 

(213)  643-1880 

833-1880 


Mr .  Keith  Karn 
Human  Factors  Department 
Kodak  Park,  Building  320 
Rochester,  NY  14650 
(716)  722-6968 
356-4157 


Dr.  Frederixk  A.  Muckier 
Essex  Corp.  (Div-21  APA) 
2135  Hartford  St. 

San  Diego,  CA  92110 
(619)276-6905 
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LT  COL  Mark  Nataupsky 
NASA-Langley  Research  Center 
M.S.  156A 

Hampton,  L»A  23665-5225 
<804)  865-3621 


Dr.  Stephen  M.  Rok i ck i 
HQ  AF0TEC/0AH 

Kirtland  AFB ,  NM  87117-7001 

(505)  844-9606 

244-9606 

Dr.  Hendrick  W.  Ruck 
AF  Inspection  4c  Safety  Center 
HQ  AFISC/SEL 

Norton  AFB,  CA  92409-7001 
(714)  382-3458 
876-3458 

Mr.  Eduardo  Salas,  Ph.D. 

Code  712 

Naval  Training  Systems  Center 
Orlando,  FL  32813-7100 
<305)  646-5130 
791-5130 


Dr .  William  F .  Storm 
Crew  Technology  Division 
USA F SAM/ON 

Brooks  AFB,  TX  78235-5000 

<512)  536-3464 

240-3464 

Col .  James  Wyman 

AFOTEC/DET  2 

Egl i n  AFB,  FL  32542 

<904)  882-4505 

872-4505 

Dr.  Larry  Shadow 
6545  Test  Group/TIE 
Hill  AFB,  UT  84056-5000 
<801)  777-6209 
458-6209 

Dr  .  Edw in  R .  Smoo tz 
ARI  Field  Unit 
HQ  TCATA  < PERI -SH) 

Ft.  Hood,  TX  76544-5065 
<817)  288-9118 
738-9118 

Ms.  Jan  Stol i ker 
6520  TESTG-ENAH 
Stop  239 

Edwards  AFB,  CA  93523 
<805)  277-3334 
350-3334 


Dr.  Charles  Nystrom 
HQ,  TCATA 

PERI-SH  <ATTN:  Dr.  Charles  Nystrom) 
Ft.  Hood,  TX  76544-5065 
<817)  288-9118/9222 
738-91 18/9222 
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TEST  AND  EVALUATION  SUB- TAG 
5  November  1985  (1:30  -  4:25) 
Lani  Room,  Kona  Kai,  San  Diego 


OPEN  SESSION 


1:30 

Introductory  Remarks 

Keith  Karn 
(NAVAIRTESTCEN) 

1:40 

Questionnaire  Construction 

Betina  Babbit 
(Essex) 

2:10 

Task  Analysis  System 

Roger  McTighe 
(Essex) 

2:55 

T&E  Practices;  Survey  Results 

Dave  Meister 
(NPRDC) 

3:20 

AFTI  Data  from  1153  Bus 

Manual  vs.  Automatic  Tracking 

Jan  Stoliker 
(Edwards  AFB) 

3:40 

Task  Analysis  Handbook  and 
Proposed  Mil.  Standard 

Jim  Geddie 
(TCATA/HEL) 

3:55 

Interaction  Between  Human 

Factors  Society  T&E  Tech 

Group  and  T&E  STAG 

Tom  Enderwick 
(NPRDC) 

4:10 

New  Business  (New  Chair,  Next 
Meeting,  Open  Discussion) 
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LITERATURE  SURVEY  AND  BIBLIOGRAPHY 
QUESTIONNAIRE  CONSTRUCTION 


5  November  1985 


Briefing  Presented  By: 

Dr.  Bettina  Babbitt 
Essex  Corporation 
7A1  Lakefield  Road 
Westlake  Village,  CA  91361 
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COMPUTER  ASSISTED  LITERATURE  SEARCH  AND  MANUAL  SEARCH 


OIAUNDIX 

DATABASES 


NUMBER  OF  CITINGS  ACTUAL  NUMBER  OF 

BY  DATABASE _  ABSTRACTS  IDENTTTTm 


NTIS  6,'766  32 

National  Technical  Information 
Sarvices,  U.S.  Department  of 
Commerce 

SOCIAL  SCISEARCH  1,448  25 

Institute  for  Scientific  Information 

COMPENDEX  462  0 

Engineering  Information,  Inc. 

PSYCINFO  466  38 

Aaarican  Psychological  Association 

AB  I/INFORM  2,384  51 

Data  Couriar,  Inc. 

SOCIOLOGICAL  ABSTRACTS  189  1 

MANAGEMENT  CONTENTS  3,359  2 

O.S.  POLITICAL  SCIENCE  DOCUMENTS  72  1 

Univarmity  of  Pittsburgh  Cantar 
for  Lntar national  Stadias 

HEALTH  PLANNING  AND  ADMINISTRATION  1,688  11 

O.S.  National  Library  of  Medicine 

ERIC  50  5 

Educational  Rasourcas  Information  Cantar 

Manual  Saarch  177 

Journals  and  Books 

TOTAL  _  _ 

16 , 816  343 


COMPARISON  OF  SCALE  CHARACTERISTICS 


i  Author 
i  Subjacts 

i  Number  of  Subjects 

i  Number  of  Experimental  Conditions 

•  Types  of  Conditions 

i  Number  of  Scale  Dimensions 

•  Number  of  Scale  Points 

i  Type*  of  Anchors 

i  Hypotheses  Measured 

•  Results 

•  Scale  Reliability 

•  Scale  Validity 
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Questionnaire  Items 


Open  Versus  Closed  Questions 
Balanced  Questions 
Length  and  Number  of  Items 
Order  of  Items 

Wording  of  Items  and  Tone  of  Wording 


Survey  Considerations 

Cognitive  Complexity 

Interviewing 

Training 

Age 

Consequences  of  Informed  Consent 

Ethnic  Background 

Education 

Non-Response  Rate 

Respondent  Anxiety 

Respondent  Preference  for  Format 
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SURVEY  RESEARCH  HAS  PROGRESSED  IN  DIVERGENT  LINES 
FOR  DIFFERENT  FIELDS 


•  Public  Opinion  Surveys  More  Or  Less  Dropped 
Tburstone  Scales  In  The  1930vs 


•  Likert  And  Guttman  Scales  Are  Primarily  Used 


•  Market  Research  Uses  The  Semantic  Differential 
Most  Frequently 


•  Performance  Appraisals  Have  Usually  Been 
BARS,  BOS,  BES 


#  Behavioral  Scales  Developed  For  Selection  And 
Morale 


•  Performance  Test&,  Pilot  Qualification,  Moving 
Toward  Guttman  Model,  Adaptive  Testing 
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FORCED  RESPONSE  AND  MULTIPLE  CHOICE  SCALES 
BEHA VI ORALLY  ANCHORED  RATING  SCALES  (BARS) 

Developed  to  measure  morale  for  military  units  (Motowidlo  and  Borman,  1977) 


Scale 

Point  Behavioral  Anchor 

9  When  maintenance  mechanics  found  an  error  in  their  assem¬ 
bly  procedures  on  an  aircraft,  they  told  their  platoon 
leaders  of  their  mistake  and  requested  that  the  hangar  be 
open  Saturday  and  Sunday  if  necessary  to  meet  their 
previously  promised  Monday  deliver. 

8  While  clearing  the  brush  from  an  approach  to  an  airport, 
these  dozer  operators  never  shut  the  dozer  off,  running 
in  shifts  right  through  lunch. 

•7  This  section  was  asked  to  prepare  a  set  of  firing  charts 
by  a  specific  time.  The  charts  were  finished  ahead  of 
time. 

6  Although  this  section  was  constantly  called  upon  for 

typing  tasks,  the  work  was  done  with  few  mistakes  and  on 
a  timely  basis. 

5  The  men  in  this  unit  did  not  push  for  top  performance, 
although  they  did  their  jobs  and  kept  busy. 

4  Many  troops  in  this  unit  would  leave  the  post  as  quickly 
as  possible  after  duty  hours  to  avoid  doing  any  extra 
work. 

3  The  service  section  of  a  support  unit  had  a  large  backlog 
of  equipment  needing  repair.  All  enlisted  personnel 
assigned  to  this  section  appeared  to  be  busy,  but  their 
output  was  very  low  compared  to  the  other  service  sec¬ 
tions. 

2  The  men  in  this  section  signed  out  weapons  to  be  cleaned 
but  sat  around  and  "shot  the  bull"  until  it  was  time  to 
turn  the  weapons  back  in. 

1  During  one  period  these  enlisted  personnel  slowed  their 
work  down  and  made  mistakes  that  cost  time  and  new  parts. 
They  were  working  7-day  weeks,  but  at  the  end  of  the 
period  they  were  accomplishing  only  the  same  amount  of 
work  in  7  days  that  they  had  been  accomplishing  before  in 
5  days. 
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New  Trend#  To  Increeee  Reliability  And  Validity,  And 


Reduce  Rating  Error# 


#  Expert  Judgment  Or  Respondent  Judgment  For 
Developmental  Procedure# 

#  Specified  Percentege  Level#  Of  Agreement 

#  Training  By  Amount  Of  Time  And  By  Content 

#  Training  Respondents  To  Reduce  Rating  Error# 

#  Filter  Question#  Assess  Conec loueneee  Of 
Respondents 


SCHUMAN,  H . ,  A  PRESSER,  S.  QUESTIONS  AND  ANSWERS 
IN  ATTITUDE  SURVEYS:  EXPERIMENTS  ON  QUESTipN  FORM, 
WORDING,  AND  CONTEXT.  NEW  YORK:  ACADEMIC  PRESS,  1981. 


•  Interviewer#  Should  Only  Record  A  Don’T  Know 

Response  When  A  Respondent  Spontaneously  Ststes  That. 


•  Don't  Know  Responses  Should  Not  Be  Included 
As  A  Fixed  Category. 

* 

•  Telephone  Interviewing  Or  Face-To-Face  Interviewing 
Had  No  Systematic  Effects  On  Survey  Research  Although 

Telephone  Respondent#  Avoid  Question#  That  Would 
Lengthen  The  Interview. 
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WEEKS,  M.  F.,  k  MOORE,  R.  P.  ETHNICITY-OF-INTERVIEWER 
EFFECTS  ON  ETHNIC  RESPONDENTS.  PUBLIC  OPINION 
QUARTERLY, 1981,  45,  245-249. 


•  A  DIff«r«nc*  In  Ethnicity  Between  Interviewer  And 
Respondent  Does  Not  Effect  The  Survey 
Retpomee  To  Non -Sensitive  Question*. 


FORCED  RESPONSE  AND  MULTIPLE  C3SOICX 
RECOMMENDATIONS 


•  The  langt_h  of  an  item  uy  possibly  modify  tha 
responsa  sty la.  Researchers  may  wish  to  develop 
alternate  versions  of  survey  items  of  varying  length. 

•  Prior  to  format  selection,  the  type  of  measurement 
scale  and  data  analysis  should  be  identified. 

•  No  one  scaling  format  has  consistently  been  superior 
to  another.  Rating  scales  need  to  be  evaluated  on 
other  criteria  than  number  of  scale  points,  vertical 
end  horizontal  formats,  and  unipolar  or  bipolar 
scalas  (Baltramini,  1982;  ZedacX,  Kafry,  &  Jacobs, 

1976) . 

•  Multiple— choic a  items  can  be  developed  with  measure 
higher  order  objectives  (Aiken,  1982). 

•  The  development  of  survey  items  should  include  pilot 
testing  using  open-ended  items  which  era  later  converted 
to  forced  response  end  multiple  choice  iteais  (Schuaan 
end  Pressar,  1981). 

•  Forced  response  end  multiple  choice  items  era  desired 
when  measuring  soft  data  such  as  opinions,  and  check - 
lists  ara  recommended  for  hard  data  such  as  physical 
aspects  of  a  yob  analysis  or  a  broad  generalization 
for  meas ruing  opinions  prior  to  a  later  survey 
(horoney,  1983;  Orlich,  1978). 
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BIPOLAR  SCALE  RECOMMENDATIONS 


•  Minor  violation*  in  the  technique  of  scale  development 
for  bipolar  anchor*,  such  as  quasi-polar  anchor*  and 
phrase*  for  anchors,  do  not  appear  to  threaten  the 
reliability  of  the  instrument  (Vidali,  1976;  Dickson 

&  Albaum,  1977).  Therefor*,  it  is  possible  to 
establish  nev  versions  for  bipolar  anchors. 

•  Researchers  need  to  be  careful  about  the  selection  of 
bipolar  scales  when  ■assuring  concepts  since  reliability 
of  the  scales  will  not  insure  validity.  The  sslectron 
of  bipolar  anchors  in  developing  a  new  scale  would  be 
more  useful  if  the  validity  of  the  scale  was  measured. 

•  rurther  research  is  required  in  the  development  of 
pictorially  anchored  bipolar  scales.  Nonverbal  pictorial 
anchors  would  eliminate  the  problem  of  translating  non¬ 
verbal  stimuli  to  verbal  anchors.  The  spplication 

of  pictorial  anchors  for  semi-literste  raters  may  be 
more  functional  for  that  group  of  rsters  than  having 
them  try  to  interpret  verbal  anchors. 

•  The  analysis  of  individual  items  In  bipolar  scales 
will  not  provide  sufficient  levels  of  reliability  so 
that  several  dimensions  are  required  to  be  summed. 

•  To  avoid  any  potential  socially  deslrabla  rasponse 
sats,  it  would  be  possible  to  anchor  the  bipolar  scale 
with  two  socially  desirable  anchors.  However,  this 
approach  would  violate  the  assumptions  of  bipolarity 
for  the  meaning,  intensity,  and  direction  of  the  seman¬ 
tic  specs.  Since  evidence  regarding  the  effects  of 
response  style  has  been  mixed,  it  would  be  better  to 
retain  the  bipolarity  of  the  scale. 


A  DECADE  OF  CHANGE 

SURVEYS  ARE  CONDUCTED  BY  COMPUTER 


•  Random  Digit  Dialing 


•  Computer  To  Raapondant  Via  Talaphona 


•  Raapondant  On  Computar  Tarmlnat 


WHERE  DO  WE  GO  FROM  HERE? 


•  Scale  Development  Procedures  and  Analyses 
t  Procedural  6uides  to  Item  Wording 

•  Automated  Portable  Test  Systems 
t  Cognitive  Complexity 

t  BEHAV I  ORALLY  ANCHORED  RATING  SCALES 
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JCUT  YOUR  TRAINING 
COSTS  WITH  ETAS® 


The  ESSEX  Training  Analysis  System  (ETAS) 
is  a  computer-based  data  collection  and 
report  generating  system  that  has  been 
developed  to  aid  in  the  application  of 
Training  Systems  Development  (TSD). 
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SURVEY  RESULTS 


TEST  AND  EVALUATION  PRACTICES 


PRESENTED  AT  T&E  SUB-TAG 
5  NOVEMBER  1985 
SAN  DIEGO,  CALIFORNIA 


DR.  DAVID  MEISTER 

NAVAL  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER 


H-23 


PURPOSES  FOR  WHICH  TESTING  IS  ACCOMPLISHED  RANKED  IN 
ORDER  OF  IMPORTANCE 


RANK  OF 

RANKS 

MEAN 

RANKS 

TO  COMPLY  WITH  GOVERNMENT  REGULATIONS 

8 

5.06 

TO  DETERMINE  THAT  THE  SYSTEM  CAN  BE 
OPERATED/MAINTAINED  EFFECTIVELY  BY 
PERSONNEL 

1 

1.47 

TO-DISCOVER  HUMAN  ENGINEERING  DESIGN 
INADEQUACIES 

2 

2.80 

TO  DETERMINE  WHICH  ONE  OF  TWO  OR  MORE 
DESIGN  CONFIGURATIONS  IS  MOST  EFFECTIVE 

5.5 

4.88 

TO  VERIFY  THAT  TRAINING  IS  EFFECTIVE. 

4 

4.6 

TO  DETERMINE  THAT  SYSTEM  CHANGE  MADE 
SYSTEM  MORE  EFFECTIVE 

5.5 

4.88 

TO  DISCOVER  THE  EFFECT  OF  VARIABLES 
AFFECTING  SYSTEM  OPERATION 

7 

4.94 

TO  INVESTIGATE  AND  RESOLVE  SYSTEM 
PROBLEMS 

3 

3.94 

TO  INVESTIGATE  RESEARCH  VARIABLES 

9 

8.0 

FREQUENCY  OF  METHOD  USAGE 


METHOD 

OVERALL 

PERCENTAGE 

RANK 

REPORTING 
•USED  CONSTANTLY 
OR  “VERY  OFTEN" 

TEST  PLAN 

1.5 

90 

QUESTIONNAIRES 

1.5 

91 

INTERVIEWS 

4.5 

85 

OBSERVATION 

4.5 

85 

OPINION  SURVEY 

4.5 

85 

RATING  SCALES 

4.5 

85 

CHECKLISTS 

ENVIRONMENTAL  TESTS  (e  g.,  SOUND, 

7 

65 

LIGHT,  TEMPERATURE) 

8 

45 

WALKTHROUGHS 

10.5 

36 

ACTIVITY  ANALYSIS 

10.5 

37 

COMPUTERIZED  METHODS 

10.5 

37 

AUTOMATED  DATA  RECORDING 

10.5 

37 

PHOTOGRAPHIC  TECHNIQUES 

14 

34 

TAPE  RECORDING 

14 

33 

CRITICAL  INCIDENTS 

14 

33 

ATTITUDE  MEASUREMENT 

16.5 

25 

ACCIDENT  INVESTIGATION 

16.5 

25 

PHYSIOLOGICAL  DATA  RECORDING 

18 

20 

DYNAMIC  MOCKUPS 

19 

10 

DISPLAY  EVALUATIVE  INDEX 

20.5 

5 

STATIC  MOCKUPS 

20.5 

4 
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METHOD  USEFULNESS 


METHOD 


OBSERVATION 
TEST  PLAN 
INTERVIEWS 
DYNAMIC  MOCKUP 
QUESTIONNAIRES 
CHECKLISTS 
RATING  SCALES 

AUTOMATED  DATA  RECORDING 
PHOTOGRAPHIC  TECHNIQUES 
ENVIRONMENTAL  TESTS  (e.g.,  SOUND, 
LIGHT,  TEMPERATURE) 

OPINION  SURVEY 

STATIC  MOCKUPS 

ACTIVITY  ANALYSIS 

ACCIDENT  INVESTIGATION 

COMPUTERIZED  METHODS 

WALKTHROUGHS 

TAPE  RECORDING 

CRITICAL  INCIDENTS 

PHYSIOLOGICAL  DATA  RECORDING 

ATTITUDE  MEASUREMENT 

DISPLAY  EVALUATIVE  INDEX 


METHOD 

METHOD 


CHECKLISTS 

QUESTIONNAIRES 

RATING  SCALES 

OPINION  SURVEY 

WALKTHROUGHS 

PHOTOGRAPHIC  TECHNIQUES 

INTERVIEWS 

ENVIRONMENTAL  TESTS  (e.g.,  SOUND, 
LIGHT.  TEMPERATURE) 

CRITICAL  INCIDENTS 
TAPE  RECORDING 
OBSERVATION 
ATTITUDE  MEASUREMENT 
STATIC  MOCKUPS 
TEST  PLAN 

AUTOMATED  DATA  RECORDING 
DISPLAY  EVALUATIVE  INDEX 
ACCIDENT  INVESTIGATION 
COMPUTERIZED  METHODS 
PHYSIOLOGICAL  DATA  RECORDING 
ACTIVITY  ANALYSIS 
DYNAMIC  MOCKUPS 


OVERALL  PERCENTAGE 

RANK  REPORTING 

'INDISPENSABLE' 
OR  'VERY  USEFUL' 


1 

100 

2.5 

82 

2.5 

80 

4 

76 

5 

70 

6 

67 

8.5 

62 

8.5 

60 

8.5 

61 

8.5 

62 

11 

57 

12.5 

53 

12.5 

52 

14.5 

52 

14.5 

50 

16.5 

35 

16.5 

33 

18 

28 

19 

19 

20 

13 

21 

0 

\SE  OF  USE 

OVERALL 

PERCENTAGE 

RANK 

REPORTING 

'USED  CONSTANTLY 

OR 'VERY  OFTEN' 

1 

80 

2.5 

75 

2.5 

76 

4 

66 

5.5 

63 

5.5 

62 

7.5 

59 

7.5 

57 

9.5 

42 

9.5 

42 

11.5 

39 

11.5 

39 

13 

37 

14.5 

32 

14.5 

33 

16 

22 

17 

20 

19 

16 

19 

16 

19 

15 

21 

10 
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FACTOR  CORRELATIONS 


Rho 

SIGNIFICANCE 

FREQUENCY  X  UTILITY 

.70 

.01 

FREQUENCY  X  EASE 

.62 

.01 

UTILITY  X  EASE 

.01 

NS 

SERIOUSNESS  OF  PROBLEM  AREAS  (PERCENT  REPORTING) 


VERY  SERIOUS/ 
IMPORTANT 

CRITICALLY 

SERIOUS/ 

IMPORTANT 

RANK 

MONEY  CONSTRAINS 

24 

24 

7 

SCHEDULE  CONSTRAINTS 

NEGATIVE  MANAGEMENT  ATTITUDES 

19 

66 

1 

TOWARD  HF 

19 

9 

14 

NEGATIVE  SUBJECT  ATTITUDES 

14 

9 

10.5 

INSUFFICIENT  NUMBER  OF  SUBJECTS 

38 

19 

4 

TOO  FEW  TEST  TRIALS 

HF  SUBORDINATED  TO  EQUIPMENT 

52 

24 

3 

ASPECTS 

DIFFICULTY  IN  APPLYING  PREFERRED 

43 

28 

2 

MSMT  TECHNIQUE 

52 

5.5 

CRAMPED  TEST  ENVIRONMENT 

LACK  OF  SUPPORTING  PSYCHOLOGICAL 

19 

4 

10.5 

DATA 

43 

8 

TOO  FEW  DATA  COLLECTORS 

9 

9 

15 

MEASUREMENT  METHODS  INADEQUATE 

33 

19 

5.5 

LACK  OF  NECESSARY  TECHNICAL  DATA 
EQUIPMENT  CONFIGURATION  KEEPS 

19 

19 

9 

CHANGING 

10 

20 

12.5 

REPEATED  EQUIPMENT  MALFUNCTIONS 

25 

5 

12.5 

H-26 


CRITERIA  USED  TO  EVALUATE  THE  SUCCESS  OF  A 
HUMAN  PERFORMANCE  TEST 


MEAN 

RANK 


DISCOVERED  HUMAN  ENGINEERING  DESIGN 

DISCREPANCIES  2.95 

SATISFIED  TEST/DEVELOPMENT  MANAGEMENT  4.35 

MADE  QUANTITATIVE  PREDICTIONS  OF  OPERATOR 

PERFORMANCE  3.52 

FORMAL  TEST  REPORT  PUBLISHED  5.55 

BEST  ONE  COULD  GET  UNDER  THE  CIRCUMSTANCES  3.5 

TEST  LED  TO  MORE  EFFECTIVE  DESIGN 

CONFIGURATION  1.75 

TEST  RESULTS  HAD  RESEARCH  IMPLICATIONS  6.1 


FREQUENCY  WITH  WHICH  STATISTICAL 
TECHNIQUES  ARE  USED 


VERY  OFTEN  CONSTANTLY 


FREQUENCY  MEASUREMENTS,  e  g., 
MEAN,  MEDIAN 
NON-PARAMETRIC  STATISTICS 
ANALYSIS  OF  VARIANCE  (ALL  VARIETIES) 
CENTRAL-COMPOSITE  DESIGN 
ANALYSIS  OF  CO-VARIANCE 
CORRELATION 
FACTOR  ANALYSIS 
MULTIPLE  REGRESSION  ANALYSIS 
"t* -TESTS 


USED  USED  RANK 


52% 

38% 

1 

42% 

9% 

2 

38% 

0 

3 

4% 

0 

8.5 

5% 

0 

7 

29% 

4% 

4 

4% 

0 

8.5 

10% 

0 

6 

20% 

0 

5 

AFTl/F-16  Data  From 
1553  Data  Bus 


Test  and  Evaluation 
Sub-Tag 
Nov  85 


Janice  R.  Stoliker 
6520  TG/ENAH,  Stop  239 
EAFB,  CA 
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'QNtmiE* 


ERP.6RAH  objectives 


1.  ASSESS  THE  ADEQUACY  OF  THE  PILOT-VEHICLE  INTERFACE  (PVI) 

2-  IDENTIFY  AND  DEVELOP  THE  METHODOLOGIES  AND  TEST  INSTRUMENTATION  FOR  DETERMINING 
PILOT  PERFORMANCE  AND  WORKLOAD  ASSOCIATED  WITH  AFTI/F-16  TECHNOLOGIES 

3-  QUANTIFY  WORKLOAD  BY  ASSESSING  PILOT  PERFORMANCE  IN  A/A  AND  A/6 
TARGET  TRACKING  IN  AUTOMATED  AND  MANUAL  MODES 


TEST  METHQDOLQ6Y 


•  ESTABLISH  CONTROL  AND  PERFORMANCE  PARAMETERS  FOR  MANUAL 
AND  AUTOMATED  TARGET  TRACKING 

•  BIOMED  INSTRUMENTATION 

•  1553  DATA  BUS 

•  SWAT  SCORES 


CONTROL  ANN  PERFORMANCE  PARAMETERS  EM  AZA  AM  AZ£  TRACKING 
AUTOMATED  Mil  MANUAL 


CONTROL 

PITCH 

ROLL 

THROTTLE  TWIST 
AZ.  EL  BIAS  COMMAND 

*  AUTOMATED  DELIVERIES 


PERFORMANCE 

DISTANCE  OF  PIPPER  FROM  TARGET 

AIRSPEED 

ALTITUDE 


-  PILOT  BLENDING  WILL  BE  EVALUATED 
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BIOMEDICAL  INSTRUMENTATION 


*  DESIGNED  AND  BUILT  BY  USAFSAM 

•  CURRENTLY  COLLECTING  EKG  DATA 

HEART  RATE 
HEART  RATE  VARIANCE 

*  BIOrtED  BOX  CAN  BE  MODIFIED  FOR: 
EM6 
EE6 

EYE  BLINK 


ELECTROCARDIOGRAM  ELECTROOE  PLACEMENT 


PILOT  SEAT  INTERFACE 


BIOMEDICAL 

LEAD 


RIGHT  HAND  CONSOLE 
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1551  MIA  m 


SllfflMY 


•  AFTI/F-16  HI6H  WORKLOAD  ENVIRONMENT 


•  DETERMINE  PILOT  PERFORMANCE  AND  ASSESS  WORKLOAD  THROU6H  THE  USE  OF: 
CONTROL  AND  PERFORMANCE  PARAMETERS 
BIOMEDICAL  DATA 
SWAT  SCORES 
1553  DATA  BUS 
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INTERACTION  BETWEEN  T&E  SUB-TAG  AND  THE  HUMAN 
FACTORS  SOCIETY  TEST  AND  EVALUATION  TECHNICAL  GROUP 


PRESENTED  AT  T&E  SUB-TAG 
5  NOVEMBER  1985 
SAN  DIEGO,  CALIFORNIA 


MR.  TOM  ENDERWICK 

NAVAL  PERSONNEL  RESEARCH  AND  DEVELOPMENT  CENTER 


HUMAN  FACTORS  SOCIETY 


•  PARENT  ORGANIZATION 

•  LOCAL  CHAPTERS 

•  TECHNICAL  GROUPS 

TEST  AND  EVALUATION  TECHNICAL  GROUP 

•  BACKGROUND 

•  HUMAN  FACTORS  SOCIETY 

•  PURPOSE  OF  TtE  TECHNICAL  GROUP 

•  RELATIONSHIP  WITH  TIE  SUB-TAG 


THE  TEST  AtB  EVAUA3TCN  TECHNICAL  GROUP 

CP 

THE  HUMW  FACTORS  aOttETt 


PURPOSE 


The  purpose  of  this  Technical  Group  is  to  pcanot m  and  foster 
ceeenrch  and  technology  to  improve  informal  and  formal ly  scheduled  tMti  xfremm 
purpose  is  to  assess  hunan  performance  with  and  without:  direct  intaraertion  with 
other  peopls,  the  envirerrnent,  systems  and/or  equipnent.  Areas  of  measurement 
include  personnel  organization,  task  organization,  skill  reguirenents ,  life 
support  systeus,  task  perforrwwe,  training  effectiveness,  biomedical  factors, 
etc.  Testing  may  involve  any  type  of  observation  and  measurement  of  task 
performance  utilizing  recxrrdlng  devices,  observer  personnel  and  subjective 
responses  during  the  design  preparation,  operation,  and  maintenance  of  systems. 


THE  TEST  AND  EVAiUATICH  TECHNICAL  GROUP 
OF 

THE  HWiN  FACTORS  30CIEIY 


AcnvniEs 


The  activities  to  be  pursued  by  the  Technical  Group  include  i 

o  Participating  in  the  Hunan  Factors  Society  Annual  Meeting  technical 
program 

o  Encouraging  and  aiding  in  the  development,  validation,  and 
application  of  methodologies,  techniques,  and  procedures  to  enhanoe 
the  reliability,  validity  and  utility  of  test  results 

o  Engaging  and  aiding  in  the  review  and  production  of  manuals, 
handbooks,  and  texts  that  contribute  to  a  huran  factors  Knowledge 
base  of  test  and  evaluation 

o  Seeking  to  promts  interest  in  and  funding  for  human  factors  test 
and  evaluation  in  industry,  govemnent,  and  academia 

o  Establishing  and  maintaining  liaison  with  other  Honan  Factors 
Society  technical  groups  and  other  professional  societies  and 
associations 

o  Reporting  of  activities  and  itans  of  interest  to  the  msiberahip, 
through  the  putolicnf  ixsi  of  a  newsletter  at  Least  twice  a  year 

o  Sponsoring  special  meetings  and  syryoeia . 
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ATTACHMENT  I 


MANNED  SYSTEM  MODELING  —  CHARTER 


AND  VIEW  GRAPHS 


November  8,  1986 


CHARTER  OF  THE  MANNED  SYSTEM  MODELLING  SubTAG 
Origin: 

Established  as  an  ad  hoc  committee  to  the  TAG,  May 
1981.  Converted  to  full  SubTAG  status  in  1983. 

Objectives : 

The  Manned  System  Modelling  SubTAG  was  established  to 
provide  a  mechanism  for  the  exchange  of  technical  infor¬ 
mation  and  the  identification  of  points  of  contact 
within  the  DoD,  civilian  agencies,  and  affiliated 
organizations,  and  to  increase  government  personnel 
awareness  of  the  state-of-the  art  in  the 
representation  of  Human  Behavior/Function  in  complex 
systems  by  formal  structures* 


Scope : 


The  technical  scope  of  the  SubTAG  includes  the  full 
spectrum  of  manned  system  simulation  tools, 
representational  formats,  and  processing  languages. 
Included  within  this  scope  are  network,  stochastic, 
control  theory,  bionic  models,  AI  data  structures,  and 
knowledge  based  machine  representation  formats. 
Application  areas  include  individual,  team,  and 
integrated  multi-weapon  system  problem  areas. 

Membership  Policy: 

Open  to  any  DoD,  NASA,  other  federal  agencies,  and 
Technical  or  Industrial  representatives, (by  invitation), 
working  in  the  area  of  manned  system  simulation  as 
defined  above  and  consistent  with  the  policies  of  the 
DoD  HFE  TAG. 

Chairperson : 


Chairpersons  are  to  be  elected  from  the  active 
membership  of  the  SubTAG  for  a  two  year  term.  The  first 
year  will  be  as  chair  elect  and  requires  working  with 
the  current  chair,  attending  TAG  operating  board 
meetings,  and  assisting  with  the  selection  of  speakers 
in  support  of  future  meetings.  During  the  second  year, 
the  chair  elect  will  assume  full  responsibility  for  the 
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SubTAG  activities  and  will  preside  over  the  election  of 
the  next  chair  who  will  in  turn  assume  full  duties  at 
the  end  of  the  second  year.  It  is  the  intent  of  the 
rotation  sequence  that  there  will  always  be  at  least 
one  available  chair  for  all  TAG  or  SubTAG  meetings,  and 
at  least  one  year  of  experience  will  have  occurred 
before  a  chair  elect  assumes  full  responsibility. 


Meeting  Schedule : 

The  normal  meeting  schedule  will  be  coincident  with  the 
main  TAG.  This  schedule  may  be  changed  at  the 
discretion  of  the  Chairman  to  include  meetings  outside 
the  TAG.  The  SubTAG  will  meet  at  least  once  a  year 
with  the  main  TAG  regardless  of  additional  outside 
meetings . 
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0  A  HUNAN  FACTURS  ENGINEERING  (HFE)  NEED  UF  AIR  FORCE  ARMAMENT  DEVELOPMENT 
( AFSC/AD) 

-  AN  HFE  DESIGN  STANDARD  THAT  INCLUDES  AWARENESS/SYSTEMS  INTERFACES 

0  IN  RESPONSE  TO  AFR  800-15,  AFSC  SUP  16.9(H)  'THE  (HFE)  OPR  HILL: 

IDENTIFY  RESEARCH  AND  TECHNOLOGY  GAPS  OR  DEFICIENCIES  THAT  CANNOT  BE 
IMMEDIATELY  ALLEVIATED' 

0  DRAFT  PRESENTATION  TO  THE  MANNED/SYSTEM  MODELING  SUB-TAG  OF  THE  DOD  HFE 
TAG 


J.  JONES,  ENGINEERING  PSYCHOLOGIST,  HFE  OPR 
AD/ENP,  EGLIN  AFB  FL  325A2-5000 
AV  872-8652,  (90A)  882*8652 
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0  AN  HFE  APPLICATIONS  PERSPECTIVE 


HFE  NEEDS  AT  PRODUCT  DIVISIONS  COULD  BE  SOLICITED  BY  RESEARCHERS, 
AND/OR 

--  HFE  NEEDS  COULD  BE  PRESENTED  TO  THE  APPROPRIATE  DOD  HFE  SUB-TAG 
FUR  COMMENTS/EVALUATION  PRIOR  TO  FURHAL  RESEARCH  FUNDING  REQUEST. 

—  THIS  INFORMAL  TWU-WAY  INTERACTION  COULD  GENERATE/STIMULATE 
PRODUCTIVE  INNOVATION. 

-  THIS  PRESENTATION  IS  A  TRIAL  EXAMPLE,  IT  ASKS 

—  HAVE  THE  APPROPRIATE  PARAMETERS  OF  COGNITION,  TASKS,  AND 
ENVIRONMENTS  BEEN  IDENTIFIED  FOR  OSE  IN  MANNED/SYSTEMS 
MODELS  FOR  ARMAMENT  DIVISION  PRODUCTS? 


U  EXAMPLE  PRUBLEM-GKDCUS,  KCS,  AMRAAM: GULF  RANGE  UKUNE  CON1ROL  UPGRADE 
SYSTEM,  RANGE  CONTRUL  SYSTEM,  AND  AMRAAM  MISSILE* 3  NEW  INTERACTING 
SYSTEMS 

CONTROL  ROOMS  FULL  OF  CONSULES  AND  LARGE  SCREEN  DISPLAYS 

P1LUTS  FLYING  MILES  OUT  OVER  THE  GULF  FIRING  A  NEW  MISSILE  UNDER 

DEVELOPMENT 

AIRBORNE  CREW  SPOTTING  BOATS  IN  THE  AREA  AND  ACTING  AS  A  TELEMETRY 

AND  AUDIO  COMMUNICATIONS  LINK 

PEOPLE  AT  CONSOLES  FLYING  DRONE  TARGETS 

SAFETY  OFFICERS,  ANALYSTS  AND  REAL  TIME  COMPUTER  SOFTWARE  INTERACTING 
QUESTION:  HOW  DOES  HFE  MUDEL  THESE  NEW  SYSTEMS  BEFORE  CURs  TO  EVALUATE 
ALL  INTERACTIONS? 

WITH  A  MODEL  WHAT  COGNITIVE  PARAMETERS  ARE  INPUT-IO,  A  BIG  SHORT  TERM 
MEMORY,  ANY  PILOT  WILL  DO  TO  FLY  DRONE,  AN  INTUITIVE  EXTROVERTED 
INTROVERT  AS  TROUBLESHOOTER? 

WHAT  INFORMATION  AND  PRESENTATION  ALTERNATIVES  WURK  BEST  WITH  WHAT 
PEOPLE  AND  MISSION  PARAMETERSt 

HOW  DU  WE  MODEL  DECISION-MAKING,  PEOPLE  RELIABILITY,  STRESS,  WORKLOAD 
DISTRIBUTION,  ETC-,  IN  ASSOCIATION  WITH  LARGE  SYSTEMS? 
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U  AH  AWARENESS/SYSJEM  INltRFACE 

-  IS  THE  SUBSET  OF  A  MANNED/ SYSTEMS  MU  DEL,  THAT 

-  IS  ANCILLARY  TO  THE  COGNITIVE/TASK/ENVIRONMENT  MODEL,  THAT 

-  INCLUDES  BIOLOGICALLY  BIASED  AND  OTHERWISE  PREDISPUSED  INDIVIDUAL  AND 
OKUUI’  AWAHENtSS  S1YLE  RESPONSES  TO  TASK/ENVIRONMENT  PERFURMANCE 
SHAPING  FAC  1  OKS- 


U  AN  AWARENESS/TASK/ENV1RONMENI  TAXONOMY 

-  WUULD  PROVIDE  ALIGNMENT  OF  PERSONNEL  SKILL  SELECTIUN  DEVICES' TO 
NATURAL  COGNITIVE  OR  AWARENESS  CONTINUUMS,  EXISTING  MANPOWER  POOLS, 
AND  TO  DESIGN  PARAMETERS  UF  SYSTEMS. 

-  WUULD  BRUADEN  PRESENT  FOCUS  IN  COGNITIVE  STUDIES  FROM  DECISION  MAKING 
PROCESSES  INVOLVING  LONG  AND  SHORT  TERM  MEMORY,  ANALYTIC-INTUITIVE 
INDUCING  ASPECTS  UF  THE  PROBLEM  DOMAIN,  THE  MECHANICS  OF  PATTERN 
RECOGNITION  PROCESSES,  ETC-,  TO  SYSTEMATIC  BIASES  OF  ANTICIPATION, 
BELIEFS,  VALUES,  MOTIVATIONS,  NEED  STRUCTURE,  ETC.,  AND  THEIR  EMOTIVE 
POWER  TU  BIND  OR  REPEL  PERSONNEL  TYPING  PARAMETERS  IFFECTING 
EFFICIENCY,  RELIABILITY,  MOTIVATION,  LEARNING,  ETC. 


0  AN  AWARENESS/SYSTEM  INTERFACE  MODEL 

-  WOULD  PROVIDE  A  TAXONOMY  UF  TASK  AND  WORK  PLACE  CHARACTERISTICS  THAT 
SENSITIZES  AWARENESS  ELEMENTS  IN  A  TAXONOMY  OF  NATURAL  CONTINUUMS 
SIGNIFICANT  TO  INDIVIDUAL  DIFFERENCES,  E-G.  SENSORY-MOTOR  TASK  ELEMENT 
THAT  SENSITIZES  A  SPECIFIC  REGION  OF  A  SENSORY  -  INTERNAL  OR  CONCRETE- 
ABSTRAL’T  AWARENESS  CONTINUUM  THAT  ALSO  ALIGNS  AND  DEFINES  NATURAL 
PERSUNNEL  -TYPE-  PREFERENCES  IN  TASK/ENVIRONMENT  CHARACTERISTICS- 


APPROACH 


0  REVIEW  LITERATURE 

-  DISTILL  DUD  COMMON  HFE  AWARENESS/TASK/ENVIRONMENT 
AND  PERFORMANCE  SHAPING  FACTOR  MUDEL  NEEDS 

-  DISTILL  A  CONSENSUS  TAXONOMY  OF  AWARENESS,  TASK 
AND  ENVIRONMENT  PARAMETERS 

-  EVALUATE  MEASUREMENT  TECHNOLOGY  FOR  APPLICABILITY 
TU  THE  PARAMETERS 

-  EVALUATE  APPLICABLE  1HEURY  AND  ADVANCE  APPARENTLY 
NEEDED  CHANGES 

-  DEVELOP  MODEL  SPECIFICATION  CONSISTENT  WITH 
DISTILLED  PARAMETERS,  MEASUREMENT  TECHNOLOGY  AND 
THEURY 
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CURRENT  STATUS 


0  REVIEWED  APPHUX 1MAT ELY  200  D0CUMEN1S 

0  ItNTATIVE  PRIMARY  CONTINUUMS!  SENSORY-INTERNAL,  SELF-OTHER 
AND  TASK/ENVIRONMENT  CHARACTERISTICS  ALONG  NATURAL 
SENSITIVITIES  10  PHYSICAL,  LOGICAL,  SOCIAL,  AND  SITUATIONAL/ 
EMOTIONAL  INVARIANTS 

0  VOCAL  SOUND  PATTERNS  IS  THE  ONLY  PHYSIOLOGICAL  MEASUREMENT 
FOUND  SENSITIVE  TO  ALL  THE  PARAMETERS  THAT  IS  UNOBTRUSIVE, 
WITH  SUFFICIENT  VARIABILITY  RANGE,  UNIVERSAL  APPLICABILITY, 
AND  RAPID  RESPONSE 

0  VOCAL  DATA  ALSU  PERTINENT  TO  MENTAL  WORKLOAD  AND  GENERAL 
COGNITIVE  MODELS 

0  AVAILABLE  VOLAL/AWARENESS  DATA  APPEARS  1U  CONFIRM  A 

CONSISTENT  1HEUKY  WITH  A  HEAVY  SUC IUBIULUGICAL  EXTENSION 

0  DRAFT  TECHNICAL  REPORT  AVAILABLE  BY  END  OF  1985 


0  INVESTIGATOR'S  RELATED  EXPERIENCE/PUBLICATIONS 

5  YEARS  HFE,  5  YEARS  R&M,  10  YEARS  COMPUTER  MODELING 
5  YEARS  MATHEMATICIAN,  2  YEARS  NAV  AVIATION  ELEC  MAINT 
BA  MATH,  MA  PSYCHOLOGY 

A  COMPARISON  OF  THE  FREQUENCIES  OF  THE  PEAK  ENERGY  OF  THE  FIRST 
FORMANT  (FI)  OF  THE  SPOKEN  VOWEL  A  FOR  TWO  OCCOPAT I ONS-PKESENTED  AT 
THE  FALL  ACOUSTICAL  SOCIETY  OF  AMERICA  MEETING  1979 
VOCAL  DIFFERENCES  BETWEEN  MEMBERS  OF  TWO  OCCUPATIONS: 

AN  EXAMPLE  OF  POTENTIAL  VOCAL/MENTAL  RELATIONSHIPS  THAT  MAY  AFFECT 
VOICE  MEASUREMENT  OF  PILOT  MENTAL  WORKLOAD  AD-TR-80-57,  1980 
US  PATENT-ORAL  SOUND  ANALYSIS  METHOD  AND  APPARATUS  FOR  DETERMINING 
VOICE,  SPEECH  AND  PERCEPTUAL  STYLES,  1984 

SOUND  LANGUAGE  AND  AWARENESS  PSYCHOLOGY,  THE  SANBIRD  PUBLISHING 
GROUP,  1985,  P-0.  BOX  1257,  SHALIMAR  FL  32579  (BOOK) 
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TYPES  OF  MODELS  USED  AT  AAMRl/HE 


•  CONTROL  THEORETIC  MODELS 

•  TASK  NETWORK  MODELING 

•  RASMUSSEN'S  FRAMEWORK 

•  BLACKBOARD  MODELING 

•  KNOWLEDGE  BASED/EXPERT  SYSTEM  MODELS 

AIR  FORCE 

AEROSPACE  MEDICAL  RESEARCH  LABORATORY 


MANNED  THREAT  QUANTIFICATION  PROGRAM 


THREAT 

ENVIRONMENT 


THREAT 

SYSTEM 


HUMAN  PERFORMANCE  INTERACTIONS 
IN  GROUND-AIR  ENGAGEMENTS 


AIRCREW  BEHAVIOR  THREAT  CREW  BEHAVIOR 


M 

KNOWLEDGE-BASED 

KNOWLEDGE-BASED 

RULE-  BASED 

<, - > 

RULE-BASED 

INTERACTIVE 

GAMING 

SKILL-BASED 

SKILL-BASED 
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IDEF/SAINT  METHODOLOGY  PROVIDES 


•  STATIC  SYSTEM  DESCRIPTION 

•  SYSTEM  PERFORMANCE  DATA  BASE 

•  DYNAMIC  SYSTEM  DESCRIPTION 


IDEF/SAINT 

A  POWERFUL  COMBINATION 


THE  IDEF/SAINT  SYSTEM  CAN  BE  USED  FOR 

•  M-M  SYSTEMS 

•  REQUIREMENTS  DEFINITION 

•  LARGE  COMPLEX  SYSTEMS 

PROVIDES  BETTER 

•  UNDERSTANDING 

•  COMMUNICATION 

•  VALIDATION 

•  MANAGEMENT 

•  ANALYSIS 

•  STANDARDIZATION 

•  DISCIPLINE 
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IDEF  MODEL 
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CREW  PROCEDURES  MODEL 


•  EXTENSION  OF  ESTIMATION  THEORY  AND  CONTROL 
MODELING 

•  HUMAN  DECISION  MAKING  BASED  ON  “SCRIPTS” 

•  RESULTS  PROMISING  FOR  PREDICTING  RULE  BASED 
PERFORMANCE 

•  GOOD  STRUCTURE  FOR  SOVIET  CENTRALIZED 
DECISION-MAKING 


A  MULTI-LEVEL  PROCESS  MODEL  FOR 
HUMAN  PERFORMANCE 


SENSORY  INPUT  DETECTION 


TRACKING 

MODELS 


P00I 

TAC  ZINGER 


OSADS 
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TASK  ROADMAP 


INTEGRATED  ANALYSIS  TECHNIQUES 


I  AT - ►  ANALYSIS  TOOL 

1.  DESCRIBES  THE  STRUCTURE  OF  C* 1 2 3 4 5  MAN -MACHINE  SYSTEM 

2.  REPRESENT  THE  DYNAMICS  OF  HUMAN/SYSTEM  BEHAVIOR 

3.  EVALUATE  AND  PREDICT  HUMAN/SYSTEM  PERFORMANCE 


I  AT - ►  DESIGN  TOOL 


4.  ASSESS  WHICH  ASPECTS  OF  A  SYSTEM  SHOULD  BE  IMPROVED  AND 
BY  HOW  MUCH 


5.  ASSESS  WHICH  PARTS  OF  A  SYSTEM  DESIGN  SHOULD  BE  ALTERED 
TO  ACHIEVE  THE  DESIRED  RESULTS 


1-11 


•  STUDY  COMPLETED 

•  CANDIDATE  TOOLS  INIUSEIOR  REVIEWED 


■mdep 

-  DESCRIPTIVE  MATRICES 

-  PAMILY  TREES 
^JPIQARPHS 
kshor:  paradigm 

•'.COUNTING  PROCESSES 


-  8AINT 

-  FORMAL  ANALYTIC  STRUCTURES 
« EVENT  ANALYSIS 

-  ENTROPY  ANALYSIS 

-  PATA  STRUCTURES 
-'AUTOMATIC  SYSTEM  DECOMPOSiiTKM 


INTEGRATED  ANALYSIS  TECHNIQUES 

IAT 


APPROACH 

•  EMPHASIZE  THE  HUMAN  AS  AN  INTEGRAL  SYSTEM  COMPONENT 

•  DESCRIBE  THIS  SYSTEM:  STRUCTURE,  OBJECTIVES,  RESOURCES... 

•  ANALYZE  -  SYSTEM  STRUCTURE 

-  SYSTEM  PERFORMANCE  , 

•  ADDRESS  -  SYSTEM  DESIGN 

-  SYSTEM  MODIFICATION 

•  DEVELOP  A  BATTERY  OF  TOOLS,  HANDBOOK 
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mwfKH&Mitt&tUHiMi 


rt 


COPE  ACCOMPLISHMENTS 

INTEGRATED  ANALYSIS  TECHNIQUES  (I AT) 


sa  -I'trriiri—irTr.^ri  ^ 


IAT  CONCEPTUAL  FRAMEWORK 


/ 


HCI  DESIGNER'S  SUPPORT  STATION 


IAT 
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RIIPL 

RAPID  INTELLIGENT  PROTOTYPING  LABORATORY 


DEFinmon 


RIIPL 


R  apid 

I  ntelligent 
P  rototyping 
L  aboratory 


-  EASY  TO  LLAUN  AND  USE 

-  CONTAINS  USI  GUIDELINE  EXPERTISE 

-  SIMULATES  THE  USI  OF  AN  APPLICATION 

-  A  1 1  AMD  WARE  AND  SOI  TWAlir.  ENVIRONMENT 


TECHMCRL  APPROACH 


RIIPL 


I  )  EXPERT  SYSTEM  -  RIPl  Incorporates  two  expert  systems: 

•  aflycln  like  consultant  expert  system  that  responds  to  general  USI  Information 
requests  (e.g.  Tell  me  about  tabular  data  presentation)  ;  and 

•  an  evaluation  expert  system  that  Interprets  the  current  definition  of  dialogues 
and  displays  (e.g.  Use  of  yellow  on  top  of  orange  Is  not  recommended). 


2)  KNQy/LEDGE  BASE  -  RIPl  Incorporates  the  Smith  &  Moslcr  guidelines  In  three 
ways: 

•  In  a  ’help  by  example*  facility; 

•  In  the  Implementation  of  itlPl.t  l.c.  limit  the  user's  options);  and 

•  In  the  USI  expert  systems  knowledge  base  (same  rule  base  for  both  experts). 


3)  PROTOTYPE  SEMANTICS  “  111  PL  combines  a  direct  manipulation  user  Interface 
with  a  graphical  ellipsis  dlaloquc  Interpreter  to  minimize  the  prototype  and  semantic 
definition  overhead. 


4)  HARO  W AREJiUlTC  -  Use  of  the  VAX  architecture  and  the  GKS  standard  allows  a 
wide  range  of  upward  migration  paths  for  processing  power  and  display  capability. 
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PROTOTYPIIIG 


RIIPL 


MAJOR  FUNCTIONS 


RIIPL 


EXECUTIVE 

Sets  up  all  necessary  file  and  library  accesses. 

Performs  housekeeping  functions  of  deleting,  renaming  and  backing  up 
RIPL  files  and  libraries. 

PROTOTYPE  BUILD  SUBSYSTEM 

Defines  screen  contents,  interactions,  and  target  environment. 
Instruments  dialogue. 

SIMULATION  SUBSYSTEM 

Compiles  simulation  components. 

Simulates,  records,  and  measures  the  USI  design. 

USI  ADVISORY  SUBSYSTEM 

Consults  on  and/or  evaluates  design. 

TECHNICAL  LIBRARIAN 

Provides  for  browsing  of  guidelines  and  examples. 

Provides  help  on  using  RIPL. 
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INPUTS  &  OUTPUTS 


RIIPL 


INPUT: 


1.  DISPLAY  CONTENT 

2.  PHYSICAL  OPERATING  ENVIRONMENT 

3.  DIALOGUE  STRUCTURE 

4.  SIMULATION  PARAMETERS 

5.  APPLICATION  OPERATING  ENVIRONMENT 


OUTPUT: 


1.  STRUCTURED  DIALOGUE  DEFINITION 

2.  DESION  CONSULTATION 
•  3.  DESION  EVALUATION 

4.  DESION  SIMULATION 

5.  CAPTURE  OF  USER  INTERACTION  DURING  SIMULATION 

6.  INSTRUMENT  MEASURES 


TERMINOLOGY 


IR II  IP  L 


STATICS 

o  tile  -  a  rectangular  area,  lest  than  or  equal  to  a  screen  in  a  size,  having 
specific  attributes  and  contents.  A  tile  is  similar  to  the  graphics  term  - 
viewport,  except  that  there  is  no  notion  of  a  larger  virtual  screen  area 
behind  it.  Tiles  may  be  grouped  in  a  hierarchy  where  tiles  that  are 
contained  in  the  area  defined  by  larger  tiles  inherit  some  of  tiieir 
attributes.  The  attributes  of  a  tile  are  dependent  on  its  position  in  the 
hierarchy'. 

o  prosaic  -  the  contents  of  a  user  activity  screen  on  a  single  workstation 
at  any  one  time.  Each  mosaic  is  composed  of  a  hierarchy  of 
tiles. 

o  gallery  -  the  collection  of  all  the  user  activity  screens  on  all  displays  at 
any  one  time. 

DYNAMICS, 

o  thread  -  a  single  chain  of  stimulus-responses  originating  with  the 
simultaneous  activation  of  a  set  of  tiles. 
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TERM  HIERARCHY 


PIIPL 


TILE  DEFINITION 


PIIPL 


o  IDENTIFICATION  -  NAME  +  ID  NUMBER 


o  CONTENT  -  (text/table/menu/bit  map/ display  list ...) 


o  SEMANTIC  USE  -  (error,  map,  title,  status  ...) 

o  LIST  OF  ATTRIBUTE  INSTANCES  (position,  size,  border,  view,  cursor  ...) 
o  LIST  OF  CONTENT  INSTANCES 
o  LIST  OF  STIMULUS-RESPONSES 
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DIALOGUE  THREAD 


RIIPL 


Sl*rt  Lid  Til*  l 


RIPL  TECHNICAL  LIBRARIAN 


RIIPL 


o  PROVIDES  ON-LINE  DOCUMENT  ACCESS 
o  FACILITATES  RAPID  INFORMATION  SEARCH 
o  FUNCTIONS  AS  RIPL  HELP  SYSTEM 
o  FUNCTIONS  AS  USI  GUIDELINE  RETRIEVAL  SYSTEM 
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HHHDWRRE  SUITE 


RIIIPL 


Approach 


1  Select  a  human-computer  interaction  problems. 

2  Develop  a  standard  task  for  this  problem  area. 

5  Develop  several  alternative  strategies  for  solving  the 
task 

1  Develop  task  network  models  for  these  strategies. 

!>  Set  model  parameters  via  several  techniques 

fo.  Collect  laboratory  data  on  subjects  performing  the  tasks 
using  the  modeled  strategies. 

7  Compare  model  predictions  to  actual  doto. 
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HC!  DESIGNER'S  SUPPORT  STATION 


7S  7\ 
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HCI  DESIGNER  TEMPLATES 


CONFLICT  RESOLUTION  METHODOLOGY 


PROBLEM 


•  TOTAL  KNOWLEDGE  CONCEPTS 

—  MULTIPLE  KNOWLEDGE  SOURCE  TRANSFORMATION 
—  NO  SINGLE  'KNOWLEDGE  CZAR' 

—  INTEGRATION  OF  DISTRIBUTED  KNOWLEDGE 

•  CONTRADICTORY  KNOWLEDGE  -  CONFLICT 

--  CATASTROPHIC  IF  NOT  RESOLVED 
—  INCONSISTENCIES 
--  DEFOCUSSED 

•  AI  MILITARY  APPLICATIONS  (I.E.  AI  IN  THE  COCKPIT) 

"  EXAMPLE  -  COCKPIT  DECISION  AIDING 


OBJECTIVES 


•  CONFLICT  RESOLUTION  STRATEGIES  UNDERSTOOD 

—  CONCEPTS 
--  CHARACTERISTICS 
—  RELATIONSHIPS 

•  CONFLICT  RESOLUTION  METHODOLOGY  DEFINED 

•  MODELING  TECHNIOUES/TOOLS  ESTABLISHED 
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HUMAN  CONFLICT  RESOLUTION  STRATEGIES 


•  ACCUMULATE.  ACCESS.  AND  EVALUATE 

—  EXPERIMENTAL  DESIGNS 
--  CONFLICT  RESOLUTION  TASKS 
—  PROTOCOL  ANALYSIS 

•  DEVELOP  CONCEPTUAL  BASELINE 

—  OPPORTUNISTIC  PROBLEM  SOLVING  MODEL 
--  EXPERIMENTAL  RESULTS 


CONFLICT  RESOLUTION  METHODOLOGY  DEVELOPMENT 


•  DEVELOP  'CONFLICT  RESOLVER'  MODEL 

—  DOMAIN  INDEPENDENT 

—  BLACKBOARD  ARCHITECTURE  BASED  (HAYES  ROTH.  1979) 
--  LISP  PROCESSOR  ENVIRONMENT 

•  REVIEW  KNOWLEDGE  ENGINEERING  TOOLS/TECHNIQUES 

--  TOTAL  KNOWLEDGE  CONCEPT  SUPPORT 

•  DEVELOP  COMPREHENSIVE  METHODOLOGY 

-  HUMAN  STRATEGIES 
—  CONFLICT  RESOLVER  MODEL 
—  KNOWLEDGE  ENGINEERING  TOOLS 


APPLICATION 


•  A I  MILITARY  SYSTEMS 

•  POTENTIAL  USERS 

-  AI  IN  ELECTRONIC  WARFARE 
--  C3  INTELLIGENT  WORKSTATION 

-  COCKPIT  AUTOMATION  TECHNOLOGY 
—  PILOT'S  ASSOCIATE 
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ATTACHMENT  J 


SUSTAINED/CONTINUOUS  OPERATIONS  —  MINUTES 


OF  3  OCTOBER  1985 


IN  REPLY  REFER  TO- 

SGRD-UWI -C 


DEPARTMENT  OF  THE  ARMY 

WALTER  REED  ARMY  INSTITUTE  OF  RESEARCH 
WALTER  REED  ARMY  MEDICAL  CENTER 
WASHINGTON,  D.C.  20307 


20  November  1985 


SUBJECT:  Minutes  of  the  2nd  meeting  of  the  Su ta i ned/Con t i nuou s 
Operations  SUB  TAG:  3  Oct  1985 


Department  of  Defense 
Human  Factors  Engineering 
Technical  Advisory  Group 


1.  The  Su s ta i ned /Con t i nuou s  Operations  SUBTAG  held  its  second 
formal  meeting  from  0830-1230  on  3  Oct  85  in  conjunction  with  the 
Human  Factors  Society  Convention  in  Baltimore,  MD.  There  were  33 
attendees.  Incl  1  is  the  agenda  and  Incl  2  the  attendance  list. 


2.  LTC  Jerry  Krueger,  SUBTAG  chairman,  reviewed  for  the 
attendees  the  short  history  of  the  group.  Copies  of  the  approved 
charter,  select  sustained  operations  publications  and  membership 
mailing  list  were  distributed.  Krueger  announced  the  2nd 
sustained  operations  special  issue  of  the  journal  Behavioral 
Research  Methods,  Instruments  &  Computers  will  be  printed  as  the 
December  1985  issue.  Members  of  the  SUBTAG  operating  board  for 
1985-86  are: 

Army : 


Navy : 


LTC  Gerald  P.  Krueger,  WRAIR  -  SUBTAG  Chairman 
Dr.  Jan  Brecht,  ARI 

Dr.  Carl  E.  Englund,  US  NHRC 
LT  Dennis  Reeves,  NAMR 1 


Air  Force:  Dr.  William  F.  Storm,  USAF  SAM 

Michael  Frazier,  USAF  OTEC 

Ex-Officio:  Dr.  Robert  G.  Angus,  DCIEM 

Dr.  David  F.  Dinges,  Ins  of  PA  Hosp. 

3.  MAJ  Roger  Stallard  of  the  Army's  Soldier  Support  Center 
directed  the  panel  of  presentations  and  discussion  on  "Modeling 
Soldier  Performance  during  Sustained  Operations."  Speakers  were: 
Roger  Stallard,  Mr.  Walt  Morgan  of  Morgan  Management  Sys.,  and 
Dr.  Terry  Klopcic  of  the  Army's  Ballistics  Res.  Lab. 


4.  Dr.  Don  Headley  of  the  Army  Research  Institute,  who  spoke  on 
behalf  of  several  different  laboratories  (including  the  Army 
Aeromedical  Research  Lab  and  the  Institute  of  Environmental 
Medicine),  and  SFC  James  Clingan  of  the  Army  Human  Engineering 
Lab  gave  a  comprehensive  update  of  sustained  operations  testing 
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program  on  the  Army's  P^NBC^  (Psychological  <5c  Physiological 
Effects  of  NBC  on  Armor  Crews  during  Sustained  Operations). 

5.  Dr.  Sol  Jacobson  of  Morgan  Management  Systems  briefly 
described  test  efforts  with  the  Walter  Reed  Army  Institute  of 
Research  in  evaluating  the  effects  of  sleep  loss  on  performance 
of  military  policemen  during  continuous  security  missions  at 
heavy  weapons  storage  sites. 

6.  LT  Dennis  Reeves  of  the  Navy  Medical  Res.  Ins.  described  his 
lab's  work  with  Navy  SEALS  in  their  r i te-of-passage  "hell-week" 
and  some  work  with  the  Marines  in  cold  weather  operations  off  the 
Northern  coasts  of  Europe. 

7.  Mike  Frazier  described  potential  applications  of  sustained 
operations  research  work  to  Air  Force  Operational  Test  and 
Evaluation  of  many  developing  Air  Force  systems. 

8.  Copies  of  most  of  the  speakers'  presentation  materials  are 
inclosures  4-9.  Presentations  by  SFC  Clingan  and  Dr.  Jacobson 
were  preliminary  reports.  Materials  from  their  talks  will  be 
available  only  after  final  reports  are  complete. 

9.  LTC  Krueger  led  the  group  in  discussion  of  3  issues: 

1) Formation  of  a  committee  to  prepare  tri-service  surrmary 
papers  of  major  sustained  operations  issues  for  the  1990s  and 
beyond,  and  to  delineate  military  and  research  needs.  Some  of 
the  material  to  be  prepared  should  be  in  the  form  of  an  update  of 
the  Woodward  &.  Nelson  "User  Oriented  Review..."  done  for  TTCP  in 
1974.  The  nucleus  of  the  corrmittee  revolves  around  SUBTAG 
opearating  board  members.  The  corrmittee  hopes  to  have  some  draft 
papers  ready  for  review  by  mid-1986. 

2 )  I  den t i f i ca t i on  of  research  needs;  funding  programs  to 
support  work  in  the  area,  and  determination  of  how  to  tie  them 
together.  Also,  determination  of  appropriate  roles  for  non-DOD 
researchers,  defense  contractors  etc.  In  the  Army  we  identified 
the  on-going  program  at  WRAIR  and  the  P^NBC^  program.  In  the 
Navy,  the  program  on  sleep  loss  etc.  at  the  Navy  Health  Research 
Center.  In  the  Air  Force,  the  occasional  work  at  the  School  of 
Aerospace  Medicne.  If  there  are  other  specifically  funded 
programs  for  research  on  sustained  operations  in  any  of  the 
services,  they  are  not  so  clearly  known  in  the  SUBTAG. 

3) Future  meeting  plans:  The  3rd  meeting  of  the  SUSOPS 
SUBTAG  will  be  held  in  conjunction  with  the  DOD  HFE  TAG  at  the 
Kennedy  Space  Center,  Florida  in  mid-May  1986. 

The  4  th  mee  ting  is  planned  as  an  all-day  workshop  in 
Washington  DC,  about  21  or  22  August  1986,  followed  by  a  2-hour 
American  Psychological  Associat ion  (APA)  symposium  on  sustained 
operations  and/or  sleep  deprivation  to  be  held  during  the  87th 
annual  APA  convention  in  Washington  DC  22-26  August  1986. 
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A  call  for  abstracts  of  papers  to  be  pre  sen  tetd -wi  1  1  be 
issued  in  December.  The  deadline  for  submission  of  abstracts  for 
the  APA  symposium  on  sustained  operations  and/or  sleep 
deprivation  is  13  Jan  86 ;  and  30  May  86  for  presentation  at  the 
August  SUS  OPS  workshop.  As  has  been  our  custom,  we  hope  to 
publish  as  many  of  the  papers  as  we  can  in  the  open  scientific 
literature,  and  as  a  minimum  at  least  in  a  set  of  proceedings  for 
the  meetings.  Thus,  if  you  intend  to  present  a  paper,  you  should 
plan  accordingly. 

10.  At  the  15th  meeting  of  the  DOD  HFE  TAG,  held  5-7  Nov  85  in 
San  Diego,  the  Chairman  distributed  copies  of  the  22  Oct  85 
Office  of  the  Undersecretary  of  Defense  for  Research  and 
Engineering  (OUSDRE)  Memorandum  for  Assistant  Secretaries  of 
Research  commending  the  TAG  for  our  tri-service  coordination  of 
research  and  exchange  of  technical  data  (Incl  3).  Some  of  you 
may  want  to  pass  a  copy  of  this  correspondence  on  to  members  of 
your  chain  of  command  as  helpful  feedback  recognizing  the  merits 
of  our  work  at  the  highest  of  DOD  levels. 

11.  LTC  Krueger  gave  an  oral  report  on  the  2nd  meeting  of  the 
SUS  OPS  SUBTAG  to  the  DOD  HFE  TAG  membership  on  7  Nov  85. 

Several  members  of  our  SUBTAG,  including  Air  Force  rep  Bill  Storm 
and  Navy  Rep  Carl  Englund  were  in  attendance. 


9  Incl  -as 


GERALD  P.  KRUEGER,  Ph.D. 
LTC,  US  Army 
Chairman,  SUB  TAG  on 
Sustained/Continuous  Ops 
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Agenda  for  2nd  Meeting  of  vthe  DOD  HFE  SUB  TAG 
on  Sustained/Continuous  Operations 
3  October  1985,  Baltimore,  Maryland 


0830  Introductions  &  SUB  TAG  Background 


0845  Modeling  Soldier  Performance  During 
Sustained  Operations  for:  Training, 
Combat  Developers,  &  Human  Engineers 
Di  scussants : 

Mr.  Walt  Morgan  of  Morgan  Mgmt  Sys. 
Terry  Klopcic,  Army  Ballistics  Lab 

0945  Coffee  Break 


LTC  Jerry  Krueger 
Wa  1 te  r  Reed 

MAJ  Roger  Stallard 
Soldier  Support  Ctr 


1000  Sustained  Operations  in  a  Chemical 
Batt lef ield  Protect ive  Posture: 

The  Army's  PZNBC2  Field  Test  Experiences 


1030  IRONMAN  -  MOPP  IV  Endurance  in  Tanks 


1045  Sustained  Operations  During 

Security  of  Special  Munitions 

1100  Marine  SUS  OPS  in  Cold  Weather 

1115  SUS  OPS  &  Air  Force  Operational  Testing 


Dr.  Don  Headley 
USA  Res  Inst. 


SFC  James  Clingan 
USA  Human  Eng  Lab 

Dr.  Sol  Jacobson 
Morgan  Mgmt  Sys 

LT  Dennis  Reeves 
Nav  Med  Res  Ins 

Michael  Frazier 
USAF  T&E  Ctr 


1130  Planning  Corrmittee  Business:  LTC  Krueger 

a)  Committee  to  prepare  DOD  Summary  Papers 
of  major  issues  for  the  1990's  &  beyond: 
military  needs;  research  issues/needs. 

b)  Identification  of  Research  Needs/Funding, 

Plans  to  tie  the  two  together  -  the  roles  of 
non-DOD  researchers,  contractors  etc. 

c)  Future  -  meetings,  publication  outlets  etc. 

1230  Meeting  Adjourns 

1235  Operating  Board  &  Others  Go  to  Working  Lunch  Together 


Incl  1 
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Name 


Attendance  Roster  3  Oct  1985 
SUSTAINED/ CONTINUOUS  OPERATIONS  SUB  TAG 


Name 

Gerald  P.  Krueger 

Daniel  P.  Redmond 
Stephen  C.  Vance 

Maria  L.  Thomas 

Organ i za  t i on 

Walter  Reed  Army  Ins.  of  Research 

•I  II  II  II  II 

•1  It  II  It  It 

tf  It  II  II  II 

William  Kok i nak i s 

J.  Terrance  Klopcic 

US  Army  Ballistics  Res.  Lab 

It  ft  0  II  It 

Roger  Stal lard 

Samuel  K.  Rock 

US  Army  Soldier  Support  Center 

It  It  II  II 

Karl  3.  Gunze  lmon 

US  Dept  of  Army,  ODCSRDA 

Gary  M.  Bratt 

HQ,  USA  Medical  R&D  Command 

David  G.  Graham 

USA  Armor  6c  Engineer  Board 

Donald  Haggard 

Donald  Headley 

Matilda  Reede  r 

US  Army  Research  Institute,  Ft.  Knox 
US  Army  Research  Institute,  Alex  VA 

US  Army  Research  Institute,  Ft.  Rucke 

Barbara  L.  Bell 

US  Army  Natick  RflcD  Center 

Jarod  B.  Jobe 

HQ,  USA  TRADOC 

C.  Thomas  Bennett 

USA  Combat  Developments  Exp  Ctr. 

Jame s  Cl i ngan 

USA  Human  Engineering  Lab 

Dennis  L.  Reeves 

Naval  Medical  Research  Institute 

Thomas  H.  Brace 

Naval  Air  Test  Center 

Sam  Sch  i  f  1  e  1 1 

USAF  School  of  Aerospace  Medicine 

Michae 1  Fraz i e  r 

USAF  Ope r  T&E  Center 

David  L.  Porterfield 

HQ,  USAF  ISC/SEL,  Norton  AFB 

Peter  Angiola 

OSD/DMMSO 

Robert  G.  Angus 

Canadian  Defence  6c  Civilian  Institute 
of  Environmental  Medicine  (DCIEM) 

Incl  2 

J-5 


Attendance  Roster  for  SUS  OPS  SUBTAG  3  Oct  85  Cont'd 


Jackson  Royal 

Dept  of  Transportation 

John  A.  Stern 

Washington  Univ  Behav  Res  Lab 

Charles  Graham 

Midwest  Res  Institute 

Walter  E.  Morgan 

Solomon  Jacobson 

Morgan  Mgmt  Sys 

M  If  II 

Elizabeth  A.  Woodward 

Old  Dominion  University 

David  F.  Dinges 

Inst  PA  Hospital  &  Univ  of  Penn 

Mark  Czarnolewski 

Not  a  f  f  i  1  iated 

OFFICE  OF  THE  UNDER  SECRETARY  OF  DEFENSE 


CNGINCERING 

(R5ATJ 


WASHINGTON  OC  20101 
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MEMORANDUM  FOR  ASSISTANT  SECRETARY  OF  THE  ARMY 

(RESEARCH,  DEVELOPMENT  AND  ACQUISITION) 

ASSISTANT  SECRETARY  OF  THE  NAVY 

(RESEARCH,  ENGINEERING  AND  SYSTEMS) 

ASSISTANT  SECRETARY  OF  THE  AIR  FORCE 
(RESEARCH,  DEVELOPMENT  AND  LOGISTICS) 

SUBJECT:  Department  of  Defense  Human  Factors  Engineering 

Technical  Advisory  Group  (DoD  HFE  TAG) 

I  am  pleased  at  the  response  of  the  Services  to  the  DoD  and 
Congressional  guidance  on  implementing  the  DoD  HFE  TAG.  The  HFE 
TAG  was  implemented  by  a  Memorandum  of  Understanding  signed  oy 
the  Assistant  Secretaries  of  the  Services  in  November  1976.  The 
TAG,  in  its  nearly  nine  years  of  existence,  has  been  a  productive 
vehicle  for  exchanging  technical  data  and  coordinating  research 
of  the  UoD  laboratories,  the  National  Aeronautics  and  Space 
Administration,  and  the  Federal  Aviation  Administration. 

Examples  of  concrete  results  attained  by  the  HFE  TAG  include: 

o  Inter-Service  coordination  in  the  following  HFE  technical 
areas : 


Controls  and  Displays 

Aviation  Personnel  Screening  and  Prediction 
Logistics 

Test  and  Evaluation 
Manned  System  Modeling 
Standardization 
Workload 

User -Computer  Interaction 
Voice  Interactive  Systems 
Technical  Society/Industry  Liaison 
Sustained/Continuous  Operations 
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o  The  development  of  an  HFE  practitioner  training  course 
implemented  by  all  military  Services. 

o  The  cooperative  compilation  and  dissemination  of  research 
pertinent  to  the  hardware  and  system  design  process. 

o  The  regular  review  and  update  of  military  specifications, 
standards  and  design  documents  by  the  services  and  industry. 

I  encourage  the  Services  to  continue  their  vigorous  support 
of  the  HFE  TAG  and  other  similar  inter-Service  research 
coordinating  groups. 

Please  convey  our  gratitude  through  the  present  Service 
coordinators  for  the  HFE  TAG. 

Dr.  Michael  Strub,  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences 

Mr.  Clarence  Fry,  U.S.  Army  Human  Engineering 
Laboratory 

Dr.  John  O'Hare,  Office  of  Naval  Research 
LCDR  Thomas  M.  Mitchell,  Pacific  Missile  Test  Center 

Mr.  Cy  Crites,  Present  Chairman,  Edwards  Flight 
Test  Center 

Dr.  Richard  Schiffler,  Aeronautical  Systems  Division 


Donald  I.  Carter 

Acting  Deputy  Under  Secretary 

(Research  and  Advanced  Technology) 


Army : 

Navy: 

Air  Force: 
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Editor's  note:  Due  to  the  lengthiness  of  the  briefings, 
enclosures  4-9  are  not  included  in  these  TAG  Minutes.  Copies  of 
the  visual  materials  of  the  following  presentations  may  be 
obtained  from  LTC  Krueger  or  the  TAG  Coordinator: 


Soldier  Performance  in  Combat  Models  —  R.  Stallard 


Modeling  Soldier  Performance  in  Sustained  Operations  — 

W.  E.  Morgan 

AURA:  Army  Unit  Resiliency  Analysis  Methodology  — 

J.  T.  Klopcic 

Psychological  and  Physiological  Effects  of  NBC  &  Extended 
Operations  on  Combined  Arms  Crews  —  D.  B.  Headley 

Navy  Medical  Research  Institute  —  D.  Reeves 

Approach  to  the  OT&E  of  Air  Force  Systems  Under  Sustained/ 
Continuous  Operations  —  M.  L.  Frazier 
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ATTACHMENT  K.l 


TRI- SERVICE  HUMAN  FACTORS  STANDARDIZATION 


STEERING  COMMITTEE 


VIEW  GRAPHS 
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HFS:  DR.  JULIEN  H.  CHRISTENSEN 


AGENDA 


Thirteenth  Meeting  of  the 


Tri -Service  Human  Factors  Standardization  Steer 


1.  Minutes  of  Previous  Meeting 

2.  Announcements 

a.  Correspondence  Summary 

b.  Membership  Changes 

c.  NASA  Man-Systems  Integration  Standards 

d.  Other 

3.  Status  Reports 

a.  Status  of  Projects 

b.  HFAC-0009 

c.  HFAC-0016 

d.  HFAC-0019 

e.  HFAC-0020 

f.  HFAC-0024 

g.  HFAC-0027 

h.  HFAC-0028 

i .  HFAC-0029 

j.  HFAC-A014 

k.  HFAC-F002 

l.  HFAC-F003 

m.  HFAC-N003 

n.  HFAC-N004 

o.  MIL-PRIME  Update  (if  applicable) 

p.  Task  Analysis  Update  (if  applicable) 

1.  ASSP  Update  (if  applicable) 

4.  Unfinished  Business 

a.  HFAC  Program  Plan,  R4 

b.  Life  Support  &  Biomedical  Factors  Coverage 

c.  PMTC  Review  of  1472C,  para  5.9 

d.  Fifth  Percentile  Female  in  the  Workforce 

e.  TAG  Proposals  to  the  JTCG/HFE/HMI 

5.  New  Business 

a.  HFAC  Program  Plan,  R5 

b.  Other 
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Manpower,  Personnel  and  Training  (MPT) 
Data  Requirements  Study 


FINAL  REPORT 
OCTOBER  1985 


Engineering  Practice  Study 
DoD  Standardization  Project  No.  HFAC-0009 
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MIL-STD- 1294A 
12  August.  1985 
Superseding 
MLL-STD-1294 
9  March  19S1 


MILITARY  STANDARD 
ACOUSTICAL  NOISF.  LIMITS 
IN  HELICOPTERS 


AMSC  A 32 4  2 


FSC  HFAC 
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METRIC 


DOD-HDBK-761 
28  June  1985 


MILITARY  HANDBOOK 


HUMAN  ENGINEERING  GUIDELINES  FOR 
MANAGEMENT  INFORMATION  SYSTEMS 

METRIC 


NO  DELIVERABLE  DATA  REQUIRED 

BY  THIS  DOCUMENT  „  ,  „  AREA  HFAC 


DISTRIBUTION  STATEMENT  A.  Approved  for  public  release;  distribution  is  unlimited. 


TABLE  I.  HUMAN  FACTORS  STANDARDIZATION  DOCUMENTS  * 


DESIGNATION 

DATE 

MIL-H-46855B 

31 

Jan 

79 

DI-H-7051 

1 

Jun 

79 

DI-H-7052 

1 

Jun 

79 

DI-H-7053 

1 

Jun 

79 

DI-H-7054 

1 

Jun 

79 

DI-H-7055 

1 

Jun 

79 

DI-H-7056 

1 

Jun 

79 

DI-H-7057 

1 

Jun 

79 

DI-H-7058 

1 

Jun 

79 

DI-H-7059 

1 

Jun 

79 

Amendment  1 

5 

Apr 

82 

Amendment  2 

5 

Apr 

84 

MIL-STD-740B 

13 

Jan 

65 

MIL-STD-783D 

18 

Dec 

84 

MIL-STD-1280 

28 

Jan 

69 

MIL-STD-1294A 

DI-H-7130 

DI-H-7031 

12  Aug  85 

TITLE 


HE  Req  for  Mil  Sys,  Equip  and  Facil 

Human  Engineering  Program  Plan 

Human  Engineering  Dynamic  Simulation  Plan 

Human  Engineering  Test  Plan 

Human  Engineering  System  Analysis  Report 

Critical  Task  Analysis  Report 

Human  Engineering  Design  Approach  Document-Operator 
Human  Engineering  Design  Approach  Document-Maintainer 
Human  Engineering  Test  Report 
Human  Engineering  Progress  Report 


Airborne  A  Structureborne  Noise  Meas  &  Acc  Crit  of  Shipboard  Equip 
Legends  for  Use  in  Aircrew  Stations  and  on  Airborne  Equipment 
Keyboard  Arrangements _ 


Accoustical  Noise  Limits  in  Helicopters 

Noise  Control  Measures  Report  (Helicopters) 
Noise  Level  Assessment  Report  (Helicopters) 


MIL-STD- 

1295A 

26 

Jun 

84 

MIL-STD- 

1472C 

2 

May 

81 

Notice 

1 

1 

Sep 

83 

Notice 

2 

10 

May 

84 

MIL-STD- 

1473A 

10 

Feb 

76 

Notice 

1 

29 

Jul 

83 

MIL-STD- 

1474B 

18 

Jun 

79 

DI-H-1336 

29 

Jul 

80 

Noti ce 

1 

10 

Oct 

80 

Notice 

2 

20 

Apr 

84 

DOD-STD- 

1477 

31 

May 

83 

Notice 

1 

31 

May 

83 

MIL-STD- 

1787 

10 

Dec 

84 

DOD-HDBK 

-743 

3 

Oct 

80 

MIL-HDBK 

-759A 

30 

Jun 

81 

DOD-HDBK 

-761 

28 

Jun 

85 

SAE  HIR- 

1622 

8 

Mar 

85 

SAE  J185 

-1970 

30 

Sep 

81 

HFE  Des  Criteria  for  Heli  Cockpit  Display  Symbology 
HE  Des  Criteria  for  Mil  Sys,  Equip  and  Fac 


Std  Genl  Reqts  for  Color  and  Marking  of  Army  Materiel 

Noise  Limits  for  Army  Materiel 
Report  of  Noise  Test 

Symbols  for  Army  Air  Defense  System  Displays 

Aircraft  Display  Symbology 

Anthropometry  of  US  Military  Personnel 

Human  Factors  Engineering  Design  for  Army  Materiel 


HE  Guidelines  for  Mgt  Info  Sys 


Noise  Conti  in  Fluid  Pwr  Sys  of  Marine  Veh 
Access  Sys  for  Const  &  Indl  Equip 

♦Excluding  unique  data  items  (UDIs)  and  data  items  approved  for  one  year  of  use. 
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Department  of  Defense 


HUMAN  FACTORS 

STANDARDIZATION  PROGRAM 

(HFAC) 

PLAN 

REVISION  4 

21  AUG  85 


DISTRIBUTION  STATEMENT  A  APPROVED  FOR  PUBLIC  RELEASE  ;  DISTRIBUTION  IS 

UNLIMITED 


I 


THE  OFFICE  OF  THE  ASSISTANT  SECRETARY  OF  DEFENSE 

WASHINGTON.  O  C.  ZOIOI-SOOO 


ACQUISITION  ANO 
LOGISTICS 


2  1  AUG  1985 


The  Human  Factors  (HFAC)  Standardization  Program  is  approved  for 
implementation  at  all  levels  within  the  Department  of  Defense. 

This  program  plan  is  the  fourth  revision  to  the  Human  Factors 
Standardization  Document  Program  Plan  published  19  June  1978.  As  Lead 
Service  Activity,  the  U.S.  Army  Missile  Command,  Redstone  Arsenal,  Alabama, 
coordinated  this  Revision,  as  developed  by  the  Human  Engineering  Laboratory 
Detachment  -  MICOM,  with  DoD  components  and  Industry.  This  activity  is 
responsible  for  implementation  and  continued  management  of  the  Human 
Factors  Standardization  Area. 

It  is  the  responsibility  of  each  identified  DoD  activity  to  supoort 
the  implementation  of  this  Plan  and  provide  the  resources  necessary  to 
complete  the  identified  tasks  witriin  the  indicated  milestones  as  provided 
for  under  the  Defense  Standardization  and  Specification  Program  (OSSP). 


Approved  by: 


John  A.  Mittino 

Deputy  Assistant  Secretary 

(Procuctior.  Sjrtort) 


DATE 
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MSIS  —  Purpose 


****** 


To  develop  a  set  of  man-system  integration  standards,  require¬ 
ments,  and  guidelines  for  human  presence  in  space.  They  will 
be  based  on  existing  standards,  requirements,  and  guidelines 
as  far  as  possible.  Gaps  and  discrepancies  in  existing  data  will 
be  identified  and  resolved  with  the  help  of  government/industry 
experts. 


SPACE 


STATION 


MSIS  —  Statistics 

^ mm  ****** 


Customer:  JSC  with  MSFC  and  Headquarters 

Value:  -$972,500 

Duration:  18  months,  starting  —  July  1985 

Subcontractors:  LMSC  (39%),  Essex  (8%),  GIAG  experts  (13%) 


K . 1-11 


ATTACHMENT  K.2 


TRI-SERVICE  HUMAN  FACTORS  STANDARDIZATION  STEERING 


COMMITTEE 


MINUTES  OF  7  MAY  1985 


Minutes  of  the  Twelfth  Meeting  of  the 
Tri-Service  Human  Factors  Standardization  Steering  Committee  (HFSSC) 

San  Antonio,  TX 
7  May  1985 

PURPOSE 

The  twelfth  meeting  of  the  HFSSC  was  convened  0830-1100  hours  on  7  May  1985 
in  the  Holly  Room  of  the  Holiday  Inn  Northwest  Loop  410,  San  Antonio,  TX,  in  con¬ 
junction  with  the  fourteenth  meeting  of  the  DoD  Human  Factors  Engineering  (HFE) 
Technical  Advisory  Group  (TAG).  The  purpose  of  the  meeting  was  to  address  the 
agenda  items  of  enclosure  1. 


ATTENDANCE 


Members  and  Alternates*  Present: 


Mr.  Gerald  Chaikin,  HEL  Detachment -MIC0M 

Mr.  Clarence  A.  Fry,  HEL 

Mr.. Larry  A.  Peterson,  HEL 

LTC  Gerald  P.  Krueger,  WRAIR 

Mr.  Richard  J.  Schiffler,  ASP 

Industry  Liaison  Representatives  . 


Dr.  Mark  M.  Brauer,  AIIE 

Mr.  Wolf  J.  Hebenstreit,  NSIA/Ei 

Others  Present: 

Mr.  Peter  Angiola,  DMSS0 
Mrs.  Helen  C.  Boggs,  MIC0M 
Dr.  James  C.  Geddie,  HEL 
Mr.  Dale  A.  Mahar,  PMTC 
LCDR  Tom  Mitchell,  PMTC 
Dr.  Gene  Schneider,  NWC 
LT  Ed  Trautman,  NPRDC 


CPT  Mike  Moroze,  AFSC 

Mr.  Monty  Joel ,  ESD 

Mr.  Doug  Miller,  HQ/SD  (AFSC) 

Mr.  Nat  Davis,  AFLC 

Ms.  Barbara  Wool  ford,  JSC  (NASA) 

Mr.  Ross  L.  Morgan,  UES 


Members  Absent: 

Dr.  John  L.  Miles,  Jr.,  ARI 
Mr.  Tom  Metzler,  AVSCOM 
Dr.  Heb'er  G.  Moore,  NAVMAT 
CDR  Larry  Dean,  NPRDC 
Mr.  Stephen  C.  Merriman,  NADC 
Mr.  Robert  N.  Deem,  AFHRL 
Mr.  William  C.  Herbert,  BMO 

Industry  Liaison  Representatives  Absent: 

Mr.  Maurice  A.  Larue,  Jr.,  AIA 
Dr.  Julien  M.  Christensen,  HFS 
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DISCUSSION 


1.  MINUTES  OF  PREVIOUS  MEETING 


Mr.  Chaikin  stated  that  the  third  sentence  under  paragraph  3.1  of  the  draft 
minutes  had  been  changed,  in  response  to  a  request  by  the  AIIE  representative, 
Dr.  Brauer,  to  read  "Conflict  with  the  U/CI  STAG  was  also  noted  as  was  absence 
of  the  AIIE  Liaison  Representative,  Dr.  Brauer,  who  was  attending  the  U/CI  STAG 
meeting."  Mr.  Chaikin  noted  that  since  no  other  changes  had  been  requested  as 
result  of  mailing  of  the  draft,  it  had  been  approved  21  March  1985. 

2.  ANNOUNCEMENTS 


a.  Correspondence  Summary.-  A  summary  of  incoming  and  outgoing  correspond¬ 
ence  (end  2)  was  furnished  the  attendees.  No  request  for  copies  of  non- 
distributed  correspondence  was  made. 

b.  Membership  and  STAG  Schedule  Block.-  Mr.  Chaikin  provided  copies  of  the 
current  membership  listing  (end  3)  and  noted  that  the  Navy  member  (P&T)  should 
read  "CDR  Larry  M.  Dean  (NPRDC)".  He  also  advised  that  the  HFSSC  would  meet 
during  block  "A"  sessions  of  each  TAG  meeting.  He  noted  that  the  only  current 
conflicts  seemed  to  be  with  the  Workload  STAG,  currently  chaired  by  Mr.  Metzler 
and  the  HEGED  Committee  on  which  Mr.  Chaikin  is  an  Army  representative.  The 
Sustained/Continuous  Operations  and  U/CI  STAGS,  currently  chaired  by  MAJ  Krueger 
and  Mr.  Peterson,  respectively,  have  been  al located  .time  in  the  block  "B" 
sessions.  Copies  of  the  HFSSC  meeting  attendance  record  was  furnished  (end  4). 

c.  Other.-  Mr.  Chaikin  noted  that  the  current  HFS  directory  and  officer 
listing  cites  the  HFSSC  as  an  organization  receiving  appointed  HFS  represen¬ 
tation  (end  5).  Mr.  Chaikin  introduced  Mr.  Peter  Angiola  and  Ms.  Helen  Boggs, 
representing  the  Defense  Materiel  Specifications  and  Standards  Office  (DMSSO) 
and  the  HFAC  Lead  Service  Activity  (MICOM),  respectively. 

3.  STATUS  REPORTS 


a.  Status  of  Projects.-  Mr.  Chaikin  furnished  the  attendees  with  summaries 
of  scheduled  actions,  dates  and  status  for  current  HFAC  projects  (end  6). 

b.  HFAC-0009.-  In  the  absence  of  Mr.  Merriman  and  Dr.  Miles,  Mr.  Chaikin 
reported  that  he  understood  the  only  action  remaining  is  to  put  together  the  EPS 
report  for  scheduled  June  coordination.  He  noted  that  the  EPS  recommendations 
have  already  been  included  in  the  draft  HFAC  Program  Plan,  R4. 

c.  HFAC-0016.  -  In  the  absence  of  Mr.  Metzler,  Mr.  Chaikin  reported  that 
the  completion  date  for  this  project  (Revision  of  MIL-STD-1294,  Acoustical  Noise 
Limits  in  Helicopters)  had  been  extended  to  the  fourth  quarter  of  FY85. 

d.  HFAC-0019.  -  Mr.  Peterson  reported  that  the  expected  completion  date  for 
this  project  (Engineering  Practice  Study  of  Keypad  Arrangements)  is  fourth 
quarter  of  FY85  and  that  formal  coordination  of  the  EPS  report  is  expected  next 
month. 
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e.  HFAC-0020.-  Mr.  Peterson  reported  that  expected  completion  date  for  this 
project  (Engineering  Practice  Study  of  Keyboard  Arrangements)  is  first  quarter 
of  FY86. 

f.  HFAC-0024.-  Mr.  Chaikin  stated  that  the  line  originals  of  "Human 
Engineering  Guidelines  for  Management  Information  Systems,"  now  designated  as 
DOD-HDBK-761,  are  currently  being  prepared  for  approval  and  submission  to  the 
Navy  Publications  and  Forms  Center.  Mr.  Peterson  stated  that  the  U/CI  STAG  will 
review  the  handbook  after  it  is  published  and  plans  to  organize  a  subgroup  to 
define  technical  content  of  a  revision.  At  that  point,  Dr.  Brauer  brought  up 
the  subject  of  section  5.15  of  MIL-STD-1472  and  its  relationship  to 
DOD-HDBK-761.  Mr.  Fry  noted  the  intents  of  a  standard  vs  a  handbook  and  dis¬ 
cussed  relationship  in  terms  of  binding  provisions  and  background,  guidance  and 
data.  Mr.  Peterson  advised  that  the  U/CI  STAG  also  plans  to  develop  the 
material  in  5.15  of  MIL-STD-1472  as  a  stand-alone  document.  Mr.  Chaikin 
concluded  the  discussion  by  advising  Mr.  Peterson  that  the  HFSSC  would  look  to 
the  U/CI  STAG  to  serve  as  agent  for  revision  of  DOD-HDKB  761  as  soon  as  possible 
after  publication  of  the  initial  issue  and  to  serve  as  agent  for  converting  5.15 
of  MIL-STD-1472C  to  a  stand-alone  standard  at  such  time  as  the  U/CI  STAG  con¬ 
siders  appropriate. 

g.  HFAC-0025.-  Mr.  Chaikin  reported  that  this  project,  completed  18  Dec  84, 
merely  transferred  MIL-STD-783D  (Legends  for  Use  in  Aircrew  Stations  and  on 
Airborne  Equipment)  to  the  HFAC  Standardization  Area. 

h.  HFAC -0026.-  Mr.  Chaikin  reported  that  this  completed  project  resulted  in 
an  acceptance  notice,  issued  8  March  1985,  on  SAE  HIR  1622,  Noise  Control  In 
Fluid  Power  Systems  of  Marine  Vehicles,  and  that  the  military  coordinating  ac¬ 
tivity  was  NAVSEA. 

i.  HFAC-0027.-  Mr.  Chaikin  announced  that  an  RFP  for  a  Human  Engineering 
Procedures  Guide  was  recently  released. 

j.  HFAC-A014.-  Mr.  Chaikin  reported  that  this  project  to  update  the  Carbon 
Monoxide  exposure  and  selected  other  provisions  of  MIL-HDBK-759A  via  a  change 
notice  would  begin  presently. 

k.  HFAC-F002.-  Mr.  Chaikin  stated.  In  view  of  comments  on  proposed 
MIL-STD-MxX,  Human  Factors  Engineering  and  Management  of  ICBM  Systems,  at  the 
last  HFSSC  meeting,  that  he  had  written  to  the  Lead  Service  Activity  recom¬ 
mending  that  any  requested  MIL-STD  number  not  be  assigned  and  advising  that  the 
twelfth  meeting  of  the  HFSSC  would  consider  cancellation  of  the  project  or  some 
other  proposal  fdr  handling  the  material  that  appears  In  the  draff, t  standard. 
Since  Mr.  Herbert  could  not  attend  the  meeting,  Mr.  Chaikin  requested  that  the 
Issue  be  deferred  to  the  next  meeting.  The  members  and  industry  liaison  repre¬ 
sentatives  concurred. 

l.  HFAC-F003.-  Dr.  Schiffler  stated  that  the  Arn\y  reportedly  signed  off  on 
using  MIL-STD-1787  vice  MIL-STD-884,  thereby  making  It  a  coordinated  MIL-PRIME. 
Mr.  Chaikin  stated  he  was  not  aware  of  such  concurrence,  assumed  that  if  such 


concurrence  was  given  that  it  came  from  AVSCOM  in  view  of  airborne  application, 
but  could  not  check  this  assumption  with  Mr.  Metzler  who  was  unable  to  attend. 
(The  information  was  provided  to  Mr.  Metzler  at  the  conclusion  of  the  meeting.) 

m.  HFAC-N003.-  Mr.  Chaikin  reported  that  this  project  was  a  new  start  on 
what  was  previously  HFAC-N001. 

n.  HFAC-N004.-  Mr.  Chaikin  reported  that  this  project  was  a  new  start  on 
what  was  previously  HFAC-N002. 

o.  MIL-PRIME-Update.-  Dr.  Schiffler  described  an  ambitious  schedule  of 
briefings  on  the  MIL-PRIME  program  to  military  and  industry  audiences  by  ASD's 
Support  System  Engineering  technical  director.  He  also  announced  the  recent 
publication  of  MIL-STD-1787  and  prospective  publication  in  June  and  July  of 
MIL-PRIMEs  on  Aircraft  Lighting  and  Aircraft  Audio  Systems,  respectively. 

Dr.  Schiffler  described  the  work  on  the  human  engineering  and  U/CI  MIL-PRIMES. 

Mr.  Fry  asked  if  ASD  was  trying  to  sell  the  MIL-PRIME  program  to  other  organiza¬ 
tions  and  services  and  Dr.  Schiffler  responded  that  they  were.  Mr.  Davis  asked 
if  the  briefing  could  be  given  to  the  HFSSC.  Dr.  Schiffler  stated  that  he'd 
check  to  see  if  an  1/2-hour  briefing  could  be  arranged.  Mr.  Fry  expressed  concern 
about  the  MIL-PRIME  philosophy.  Mr.  Chaikin  asked  if  it  would  be  useful  to  set 
aside  20-30  minutes  after  the  briefing  to  discuss  these  concerns,  rather  than 
covering  them  at  this  time,  and  Mr.  Fry  and  others  agreed.  At  the  close  of  the 
meeting.  Dr.  Schiffler  provided  a  hard  copy  set  of  MIL-PRIME  briefing  viewgraphs 
(end  7). 

p.  Task  Analysis  Update.-  Dr.  Geddie  reported  on  the  "offline"  effort  by 
HEL  and  the  HFTE  STAG  to  structure  material  suitable  for  use  as  a  possible 
MIL-STD  on  Task  Analysis.  HEL,  he  noted,  has  contracted  this  work  to  Battelle 
Columbus  Lab  and  is  keeping  it  oriented  toward  Army  T&E  for  the  time  being.  He 
advised  that  the  T&E  STAG'S  consensus  is  that  the  effort  is  on  track  and 
announced  that  the  first  draft  should  be  available  at  the  next  HFTE  STAG 
meeting.  Dr.  Geddie  understands  that,  at  the  appropriate  time,  the  HFTE  STAG 
will  decide  if  the  draft  is  suitable  for  establishing  a  formal  standardization 
project.  The  document  is  seen  as  containing  four  sections:  1.  Requirements, 

2.  Tailoring  Guide,  3.  DIDs  and  4.  Methodology  Guidance.  He  remarked  that 
structuring  of  task  analysis  DIDs  would  seem  to  involve  some  change  to 
DI-H-7055,  7056  and  7057.  He  will  continue  to  keep  the  HFSSC  apprised  of  status 
and  decisions. 

q.  ASSP  update.-  Dr.  Schiffler  reported  that  the  Aircrew  Stations 
Standardization  Panel  (ASSP)  met  27-28  March;  identified  MIL-STD-203  (Aircrew 
Station  Controls  and  Displays  for  Fixed  Wing  Aircraft),  MIL-STD-250  (Aircrew 
Station  Controls  for  Rotary  Wing  Aircraft),  MIL-STD-411  (Aircrew  Station 
Signals)  and  MIL-STD-850  (Aircrew  Station  Vision  Requirements  for  Military 
Aircraft)  as  requiring  updates;  noted  that  a  MIL-STD-411  working  group  has  been 
organized;  and  that  a  draft  revision  of  MIL-STD-203  is  about  ready  for  cir¬ 
culation.  Dr.  Schiffler  also  advised  that  he  chairs  a  Night  Vision/Aviation 
Lighting  System  Committee  which  is  developing  a  Tri-service  document  to  be 
published  as  a  military  specification  with  Navy  lead. 
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5.  NEW  BUSINESS 


a.  TAG  Proposals  to  the  JTCG/HFE/HMI Mr.  Chaikin  noted  that  his  solicita¬ 
tion  for  inputs  to  the  Joint  Technical  Coordinating  Group  for  Human  Factors 
Engineering/Human  Machine  Interface,  as  requested  by  the  TAG  representative  to 
the  JTCG/HFE/HMI  (Dr.  Geddie),  inadvertently  imposed  a  29  March  instead  of  a 
29  April  suspense,  probably  accounting  for  lack  of  response  in  time  for  this 
meeting.  Dr.  Geddie  reminded  the  committee  that  submission  through  the  TAG  of 
candidate  study,  research  or  other  efforts  appropriate  for  sponsorship  by  the 
JTCG/HFE/HMI  is  a  continuing  need.  As  such,  it  was  agreed  to  keep  the  item  on 
the  agenda. 


b.  Other.-  A  highlight  of  the  meeting  was  representation  from  both  DMSSO 
and  the  HFAC  Standardization  Project  Officer  from  the  Lead  Service  Activity — 

Mr.  Angiola  and  Ms.  Boggs,  respectively.  Mr.  Angiola  provided  a  copy  (end  9) 
of  the  current  (Jan  85)  Defense  Standardization  and  Specification  Program  Report 
to  Congress  along  with  suggestions  regarding  the  thrust  of  the  HFAC  standardiza¬ 
tion  accomplishment  reports.  Of  32  standardization  areas  only  about  a  half- 
dozen  were  covered  by  the  Technology  Standardization  Section  and  HFAC  was  one  of 
the  areas  covered.  Finally,  Dr.  Schneider  announced  that  NWC  has  recently 
developed  a  guideline  for  acquiring  desktop  computers  on  the  basis  of  human 
engineering  and  other  user-critical  features  and  announced  that  copies  of  the 
guidelines  are  available  on  request. 


9  Ends 


GERALD  CHAIKIN 
Chai rman 
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4.  UNFINISHED  BUSINESS 


a.  MIL-STD-490  Coverage  of  GAO  Target  Areas.-  Mr.  Chaikin  flpted,  from  the 
last  minutes,  that  Mr.  Heckart  had  contacted  the  preparing  activity  of  MIL-STD- 
490  to  determine  status  of  the  HFSSC-proposed  changes  and  reported  that  the 
HFSSC  recommendations  are  being  kept  on  file  pending  proposed  revision  of  or 
change  notice  action  on  the  standard.  After  a  brief  discussion — primarily 
between  Mr.  Fry,  Dr.  Brauer  and  Mr.  Hebenstreit— regarding  the  merits  of 
retaining  this  item  on  the  agenda,  a  vote  was  taken  as  a  basis  for  dropping  the 
item  (Drop:  Mr.  Hebenstreit,  Mr.  Fry,  Dr.  Schiffler,  Mr.  Chaikin.  Retain: 

Dr.  Brauer,  LTC  Krueger,  Mr.  Peterson.) 

b.  HFAC  Program  Plan,  R4.-  Mr.  Chaikin  announced  that  the  draft  was  sent 
out  for  formal  coordination,  1  May  85  with  a  10  July  suspense  and  provided 
information  copies  (end  8)  to  the  attendees.  Dr.  Brauer  requested  that  two 
additional  copies  be  sent  to  him  and  asked  that  the  next  formal  coordination 
letter  going  to  AIIE  identify  him  as  POC  to  facilitate  internal  coordination. 

Dr.  Schiffler  also  requested  an  extra  copy.  Mr.  Chaikin  agreed  to  take  care  of 
these  Items. 

c.  User/Computer  Interface.-  Mr.  Peterson  advised  the  Committee  that  the 
U/CI  STAG  is  willing  to  generate  the  stand-alone  U/CI  standard  and  the  revision 
of  DOD-HDBK-761  as  previously  mentioned.  Consequently,  this  item  was  dropped 
from  the  agenda  as  unfinished  business  but  will  be  covered,  in  the  future,  by 
status  reports  upon  establishment  of  formal  projects. 

d.  Life  Support  &  Biomedical  Factors  Coverage.-  MAJ  Krueger  stated  that  he 
would  a)  write  a  letter  to  the  Occupational  and  Environmental  Health  Lab  at  the 
School  of  Aerospace  Medicine  requesting  comments  on  the  BMO  draft  document  and 
b)  send  the  Tri -service  Technical  Group  for  MIL-STD-1472  a  copy  of  the  BMO 
draft,  redlined  to  highlight  provisions  which  can  be  considered  for  Inclusion  in 
MIL-STD-1472. 

e.  PMTC  Review  of  MIL-STD-1472C,  Para  5.9.-  Mr.  Mahar  announced  that  the 
results  of  the  Pacific  Missile  Yest  Center  review  of  the  Maintainability  Design 
section  of  MIL-STD-1472  will  be  sent  out  in  proposed  revision  form  for  review  in 
the  near  future.  He  advised  the  committee  that  a  dozen  copies  would  be  sent  to 
the  Tri-service  Technical  Group  for  MIL-STD-1472. 

f.  Fifth  Percentile  Female  in  the  Workforce.-  This  subject  was  briefly 
discussed--pr1marily  by  Mr.  Hebenstreit,  Dr.  Brauer,  Mr.  Fry  and  Dr.  Schiffler. 
Mr.  Hebenstreit  noted  that  the  most  prominent  problems  seemed  to  be  in  the 
fighter  cockpit  design.  Dr.  Brauer  agreed  and  felt  that  the 'problem  was  common 
to  aircrew  stations.  Mr.  Chaikin  asked,  In  view  of  the  aircrew  station  focus, 

If  the  ASSP  had  ever  reviewed  the  problem.  Dr.  Schiffler  responded  that  it  had 
but  never  resolved  it.  The  discussants  agreed  that  the  issue  involves  a  policy 
statement  that  cannot  be  resolved  by  the  HFSSC.  Mr.  Chaikin  suggested,  without 
dissent,  that  this  Item  be  deferred  to  the  next  meeting  since  Mr.  Herbert  was 
unable  to  attend  the  twelfth  HFSSC  meeting. 
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TRI-SERVICE  WORKLOAD  COORDINATING  COMITTEE 


—  MINUTES,  VIEW  GRAPHS 


DRAFT  MINUTES:  Sub  Technical  Advisory  Group  (SubTAG) 
Workload  Coordinating  Committee 


1.  Tne  meeting  of  the  SudTAG  Workload  Coordinating  committee  was  neld  cn 
5  November  1985  in  conjunction  with  the  Department  of  Defense  Human 
Factors  Engineering  Technical  Advisory  Grcup  meeting  at  the  Kona  Kai 
Resort,  Shelter  Island,  San  Diego,  Califcrnia.  The  session  was  chaired 
oy  Heidi  Fiedler  from  tne  Naval  Underwater  Systems  Center.  Persons  who 
attended  tne  session  are  listed  on  the  attached  roster  of  attendees 
(Enclosure  l),  along  witn  their  Tn -Service,  NASA,  or  otner  affiliation. 

2.  The  meeting  was  divided  into  three  pnases;  (a)  Announcements  and  New 
Business,  (b)  Action  Items,  and  (c)  Presentations. 

(a)  Tne  cnai rmansnip  was  rotated  from  Tom  Metzler  of  the  Army 
Aviaticn  Systems  Researcn  Command,  to  Heidi  Fiedler  of  the  Naval 
Underwater  Systems  Center,  Newport,  Rhode  Island.  Three  items  of  new 
business  were  discussed: 

1.  The  first  was  tne  possibility  of  an  extended  workload  Suotag 
meeting  in  tne  spring  of  1986.  It  was  proposed  that  this  2  to  3  day 
meeting  wculd  oe  conducted  like  a  wcrksnop  ratner  tnan  the  standard 
format  for  serial  presentations.  Through  a  vote  it  was  indicated  that 
approximately  20  people  wculd  be  interested  in  attending  sucn  a  session. 
It  was  suggested  that  tnis  meeting  be  split  up  oetween  Monday  and  Friday 
around  tne  mother  TAG  meeting  to  avoid  tne  problem  of  two  travel  requests 
oeing  submitted  in  close  time  proximity.  Another  alternative  wculd  be  tc 
hold  tne  wcrksnop  in  conjunction  with  a  national  meeting,  i.e.,  NAECON, 
HFS.  Tne  group  was  surveyed  as  to  what  would  be  most  beneficial  as  far 
as  content  in  such  a  meeting.  The  results  were: 

o  Tutorials,  not  just  paper  presentations. 

o  Discussions  on  how  to  take  HF  principles  out  cf  the  lab  and  into 

the  field. 

o  Availaole  measurement  techniques  and  new  to  use  them. 

o  How  to  validate  a  system. 

o  Nonintrusive  measures  using  the  1553  Bus. 

o  Hew  to  validate  measures. 

o  Methodologies  and  relations  to  otner  performance  measures. 

o  Workload  and  Training. 

o  Hew  do  designers  (development  w/o  research)  address  tnese  issues, 

1. e.,  allocation  of  functions. 

2.  Tne  issue  of  a  cnair  select  fer  tne  SudTAG  was  addressed.  The 
objective  is  tc  provide  support  and  oackup  to  tne  current  cnairman  and 
continuity  upon  rotation  cf  tne  cnair  position.  Tnis  position  should 
rotate  to  the  Air  Force  next,  and  anyone  currently  assuming  this  position 
would  move  into  tne  cnairmanshp  in  Novemoer  1986.  Interested  members 
were  asked  tc  repert  to  tne  current  chairperson,  H.  Fiedler. 
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3.  It  W83  suggested  tnat  the  workload  SubTAG  develop  liaisons 
witn  all  tne  otner  SubTAGS.  Many  of  the  otner  SubTAGs  (i.e.,  Controls 
and  Displays,  Modeling,  and  UCl)  are  dealing  witn  problems  and  issues 
that  are  very  closely  related  to  numan  workload.  A  liaison  system  would 
allow  for  greater  communication  between  these  SuoTAGs  than  is  pcssiole 
tnougn  tne  final  day  summary  presentations.  This  issue  will  be  pursued 
further  oy  the  chairman  of  tne  workload  SubTAG,  H.  Fiedler. 


(o)  A  draft  of  tne  Workload  Standards  Document  was  distributed  by 
G.  Reid  from  Wrignt-Patterscn  Air  Force  Base.  Tnis  document  is  evolving 
as  a  workload  subtag  product  and  consequently  should  be  thoroughly 
reviewed  oy  tne  members  of  tnis  group.  It  is  desired  to  have  feedback 
from  the  members  of  the  SuoTAG  concerning  the  basic  approacn,  tne  need 
for  tne  document,  or  any  ether  relevant  comments.  Members  are  encouraged 
to  send  tneir  replys  to  H.  Fiedler  as  soon  as  possible.  An  extensive 
review  of  tnis  document  will  be  planned  for  the  spring  meeting. 

Tne  workload  register  is  completed  and  is  currently  being  published. 
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(c)  Presentations : 


PHYSIOLOGICAL  AND  BEHAVIORAL  WORKLOAD  MEASURES 
IN  FULL-MISSION  SIMULATORS 

Dr.  Richard  Schiffler 
ASD/ENECH 

W right- Patterson  Air  Force  Base 

Dr.  Scniffler  was  presenting  for  J.  Gavem  wno  is  currently  involved 
in  this  work. 

Tne  main  efforts  cf  the  Human  Factors  brancn  at  tne  Aeronautical 
Systems  Division  are  in  tne  applications  area  not  in  tne  research  area. 
Tne  goal  is  not  to  develop  metrics  but  to  apply  them  in  support  cf 
various  weapon  systems.  Tne  main  objective  is  to  use  workload  metrics  in 
designing  and  developing  crew  stations.  Tne  Human  Factors  orancn  nas  a 
Crew  Station  Design  Facility  (CSDF)  wnere  engineering  man-in-tne-locp 
sumulaticns  address  specific  issues  (e.g.,  Displays  and  Controls  used  on 
tne  F-16). 

Dr.  Scniffler  descrioed  workload  as  a  "means  to  an  end".  The  end 
being  now  to  design  a  cockpit  to  effectively  use  the  man  in  the  system. 
The  CSDF  currently  nas  three  simulators,  an  F-16,  an  A-10  and  a  C-18 
wideoody  simulator.  These  simulators  are  used  as  engineering  design 
tcols  and  not  as  training  simulators.  Head-ups  display  capabilities  are 
also  available.  The  CSDF  is  currently  running  a  series  of  simulations  in 
support  of  several  pregram  offices. 

Some  of  tne  topics  addressed  in  tne  past  and  currently  of  concern 
are:  crew  composition  level  related  tc  reducing  the  number  of  crew 
members,  night  visual  simulations,  human-macnine  function  allocation,  and 
the  location  and  arrangement  of  controls.  Tne  branch  is  currently  very 
neavily  committed  to  supporting  tne  F-16  project  office.  Such  issues  as 
information  flow  and  formatting,  mul tifuncticn  displays,  and  automation 
are  currently  oeing  studied.  Tne  questions  to  be  assessed  are:  Does  it 
reduce  tne  workload  level?  Dees  it  modify  the  workload  level?  What  is 
necessary  is  a  tactical  evaluation  of  ooth  systems  and  subsystems  to 
thoroughly  understand  the  total  numan-machine  integration  issue. 

The  CSDF  uses  operational  crews  of  different  levels  in  tneir 
experiments.  Some  of  the  performance  measures  used  are:  aircraft 
parameters,  subjective  questionnaires,  video  taping,  target  acquisition 
data,  and  time.  Dr.  Schiffler  tnan  reviewed  the  workload  assessment 
methods  used  oy  his  branch. 

The  workload  assessment  methodologies  used  include:  task  analysis, 
direct  observation  (througn  participation  witn  C-141  and  C-135  aircraft 
flight  tests),  aircraft  parameters,  performance  data,  subjective 
questionnaires,  (e.g.,  MCH,  SWAT).  Traditionally  tne  subjective  methods 
have  provided  most  of  the  data  and  the  Human  Factors  branch  would  like  to 
use  more  behavioral  and  physiological  techniques  as  they  appear  to  have 
promise. 
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One  task  analytic  benavioral  method  used  by  tne  branch  is  tne  Task 
Analytic  apprcach.  In  tnis  apprcach  a  Datamyte  is  used  to  record  the 
time  availaoie  to  accomplisn  a  task  versus  tne  time  required.  Tne  key 
nere  is  to  identify  each  majcr  task  a  pilot  is  performing  and  record  tne 
time  spent  on  tne  task.  Tnis  metnod  nas  been  used  in  some  cf  tne  CSDF 
simulations  and  test  programs. 

Tne  task  analytic  perfcrmance  apprcach  looks  at  very  specific  things 
sucn  as  target  acquisiticn  and  navigation  and  flignt  control.  Reaction 
time  measures  and  perceptual  motor  assessments  comprise  tnis  apprcach. 

The  physiological  apprcacn  is  in  its  nascent  stages  and  tnese  techniques 
are  just  moving  into  applied  settings.  Seme  of  the  data  obtained,  in 
addition  to  performance  data,  are  pupil  dilation,  brain  activity,  voice 
analysis,  muscle  tension,  and  galvanic  skin  response. 

To  explore  further  and  to  apply  pnysiological  and  benavioral  workload 
teenniques  in  our  engineering  simulations  tne  Human  Factors  oranch  has 
signed  a  contract  with  Ergotecn  to  develop  the  behavioral  and 
pnysiological  workload  metnodologies,  including  nardware  and  software,  to 
use  in  simulations. 

This  project  is  oriented  toward  developing  nonobtrusive  techniques  to 
use  workload  metrics,  in  addition  to  other  perfcrmance  metrics,  to  assess 
various  crew  staticn  ccnfigurations.  The  initial  phase  includes  the 
determination  of  touch  capabilities,  development  cf  hardware, 
demonstration  and  testing,  design  and  fabrication,  and  instalation  in  the 
CSDF. 

The  objective  is  to  develop  techniques  that  can  oe  used  in  all  three 
simulators.  It  is  necessary  tc  have  the  flexibility  to  modify  tne 
various  pnysiological  and  performance  teenniques  for  a  particular 
activity  of  interest  (e.g.,  motor  responses). 

A  problem  that  nas  existed  in  tne  workload  arena  for  years  is  the 
development  of  global  ratner  than  specific  measures  of  crew  or  pilot 
workload.  It  is  hoped  that  tnis  project  will  provide  more  specific 
measures  of  the  different  aspects  or  dimensions  of  workload.  Workload  is 
not  a  one-dimensional  measure  but  consists  cf  varicus  components  that 
will  determine  what  the  workload  level  is  and  hew  it  varies. 

By  looking  at  tne  physiological  and  performance  measures  Dr. 

Scniffler  hopes  tnat  researchers  will  have  a  better  handle  on  not  only 
tne  overall  workload  level,  but  the  potential  for  a  given  design  to 
increase  workload  under  certain  environmental  conditions.  Tne  main 
purpose  is  tc  use  these  techniques  as  diagnostic  tools  to  provide 
information  aocut  tne  "best  designs”.  For  example,  if  you  have  several 
configurations  that  look  similar  and  in  the  first  cut  the  relative 
workloads  or  processing  times  are  tne  same,  nopefully  workload  measures 
will  be  a  good  diagnostic  tool  to  say  "under  these  conditions,  this 
configuration  is  in  fact  a  better  design." 

Dr.  Scniffler* s  CSDF  efforts  are  in  the  early  stages  of  trying  to 
apply  these  behavioral  and  physiological  techniques.  The  quest  is  for  a 
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quick,  easily  adaptable  tool  that  is  nonintruaive,  naa  a  stand-alone 
capability,  and  tnat  can  easily  be  calibrated  to  the  system. 

Intrusiveness  is  a  oig  problem  with  workload  assessment  methods. 

Hopefully  tne  result  of  this  project  will  be  a  nonintruaive  technique  to 
look  at  and  assess  design  layouts  of  particular  cockpits.  It  is 
necessary  to  take  tacnmques  out  of  lab  into  the  development  community  to 
see  how  tney  can  best  aid  us  in  tne  major  ta3k  of  cockpit  design. 


PERFORMANCE  MEASURES 


•  MISSION  LOG  SHEETS 


•  AIRCRAFT  PARAMETERS 


•  SUBJECTIVE  QUESTIONNAIRES 


•  VIDEOTAPE 


•  TARGET  ACQUISITION 


•  TIME 
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HUMAN  ENGINEERING  DESIGN  TOOL 


HUMAN  US  MACHINE  FUNCTION  ALLOCATION 

LOCATION  &  ARRANGEMENT  OF  CONTROLS  &  DISPLAYS 

CREW  TASK  LOADING 

INFORMATION  FLOW  AND  FORMATTING 

TACTICAL  EVALUATION  OF  SYSTESM/SUBSYSTEMS 

TOTAL  HUMAN-MACHINE  INTEGRATION 


WORKLOAD  ASSESSMENT  METHODOLOGY 

•  TASK  ANALYSIS 

•  Direct  observation 

•  AIRCRAFT  PARAMETERS 

•  Primary  tasks 

•  SUBJECTIVE 

•  QUESTIONNAIRES 

•  MODIFIED  COOPER-HARPER  SCALES 

•  SUBJECTIVE  WORKLOAD  ASSESSMENT  TECHNIQUE 

•  BEHAVIORAL 

•  SECONDARY  TASKS 

•  PHYSIOLOGICAL 
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PERFORMANCE  APPROACH 


FLIGHT 

CONTROL 


NAVIGATION 


BEHAVIORAL  METRICS 

•  VISUAL  STERNBERG  TASK 

•  AUDITORY  STERNBERG  TASK 

•  INTERVAL  PRODUCTION  TASK 

•  LEXICAL  ACCESS  TASK 

I  DISPLAY  MONITORING  TASK 
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PSYCHOLOGICAL  APPROACH 


Pupil 

Dilation 


Brain  v 
Activity 


Voice 


Heart 

Beat 


Gmvanic 
Sit  in 
Responsi 


Muscle 

Tension 


PHYSIOLOGICAL  APPROACH 
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PHYSIOLOGICAL  METRICS 


•  VISUAL  CORTICAL  EVOKED  RESPONSE 


•  AUDITORY  CORTICAL  EVOKED  RESPONSE 


•  EPOCH  ANALYSIS  OF  EEG 


•  EYE  BLINK  ACTIVITY 


SIMULATOR  WORKLOAD  MEASUREMENT  SYSTEM  (SWMS) 


BEHAVIORAL  AND  PHYSIOLOGICAL  WORKLOAD  MEASUREMENTS 


•  HARDWARE/SOFTWARE 


•  CONTRACT  LET  FOR  PHASE  I  JULY  1985 
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PHASE  I 


•  DETERMINATION  OF  DESIRED  TEST  CAPABILITIES 

•  DEVELOPMENT  OF  UNIQUE  HARDWARE  AND  SOFTWARE 

•  BREADBOARD  DEMONSTRATION  AND  TESTING 

•  DESIGN,  FABRICATION,  INSTALLATION,  AND  TEST 
AT  CREW  STATION  DESIGN  FACILITY  (CSDF) 

•  DOCUMENTATION  OF  HARDWARE,  SOFTWARE,  AND 
PROCEDURES 

SWMS  FEATURES 

I  QUICK  AND  EASY  ADAPTABILITY 

•  STAND  ALONE  CAPABILITY 

•  TIME-CODED  WORK  LOAD  DATA 

•  SIMPLIFIED  OPERATION 

•  MENU  OPTIONS 

•  PROMPTS 

•  EMBEDDED  ST1HULUS/RESPONSE  INTERFACES 


SWMS  ADVANTAGES 


•  STANDARDIZATION 


•  FLEXIBILITY 


GLOBAL  VS  SPECIFIC  WORKLOAD 


•  DIAGNOSTIC 
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JOINT  TACTICAL  INFORMATION  DISTRIBUTION  SYSTEM 
( JTIBS )  I  0  T  &  E 


Colonel  Jim  Wyman 

Air  Force  Operational  Test  &  Evaluation  Center 
Eglin  Air  Force  Base 


Colonel  Wyman  talked  on  tne  Joint  Tactical  Information  Distribution 
System  (JTIDS).  He  is  presently  test  director  for  tne  initial  test  and 
evaluation  of  tnis  system.  Tne  purpose  of  tnis  system  is  to  be  able  to 
accommodate  a  very  nigh  rate  of  data  transmission  while  providing  a 
secure,  anti-jam  capability  at  the  battlefield  sight. 

Enormous  quantities  of  data  are  transmitted  to  air  defense  units,  and 
tne  units  tnen  process  tne  data  resulting  in  a  cnange  in  mission 
performance.  Colonel  Wyman  explained  tnat  as  operational  testers  it  is 
tne  job  of  the  center  to  evaluate  tne  JTIDS  Class  2  terminal  and 
determine  the  cnange  in  mission  performance  when  it  is  owned,  operated 
and  maintained  by  tne  respective  services. 

This  will  be  accomplisned  by  plugging  JTIDS  into  an  operational 
environment,  not  a  laboratory  environment,  in  an  effort  to  use  it  in  as 
realistic  a  setting  as  possible.  Two  aircraft,  the  F-15A  and  F-15C  will 
be  used  for  evaluation. 

Tne  system  works  on  a  time  division  multiple  access  architecture. 

This  means  that  every  user  has  a  transmitter,  receiver  and  clock.  Eacn 
user  is  assigned  a  time  period  wnen  he  can  transit  data  and  at  all  other 
times  ne  can  only  receive  data. 

Presently  tne  pilot  gets  information  from  a  number  of  sensor  sources, 
out  anything  external  to  tne  cockpit  will  have  to  come  over  tne  UHF 
radio.  Consequently,  the  bits  and  pieces  of  information  are  all  coming 
in  serially,  and  one  has  to  stop  oefcre  tne  next  one  can  be  given.  JTIDS 
allows  all  tne  various  users,  senders  and  receivers,  to  be  able  to 
transmit  this  information  in  parallel.  The  question  is,  how  does  one 
display  tnis  information  to  the  pilot? 

Critical  in  the  display  of  tne  JTIDS  transmitted  information  is  the 
Multipurpose  Color  Display.  Tne  Multipurpose  color  display  uses  five 
colors  to  display  air  defense  information.  Essentially  this  is 
incorporating  a  wnole  new  method  of  displaying  information.  It  is  not 
new  to  tne  community  at  large,  out  it  is  new  to  the  150-200  hour  F-15 
pilot  tnat  will  be  using  it.  Tne  question  is  will  the  pilot  be  able  to 
take  this  information  and  process  it,  turning  it  into  something  useful, 
something  that  will  improve  his  mission  capabilities. 

The  test  is  a  10  1/2  month  test  broken  into  3  segments:  l) 
man-in-tne-loop  simulations;  2)  flight  testing  —  to  trap  perishible 
data;  and,  3)  operational  exercises.  Quantifiable  information  will  come 
from  tne  simulations  comparing  the  use  of  JTIDS  with  situations  where 
JTIDS  was  not  used.  Scenarios  developed  in  the  first  segment  will  be 
interwoven  through  the  second  segment.  Tne  third  segment  will  involve 
actual  mission  participation. 
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Tne  goal  is  to  determine  the  actual  contribution  of  JTIDS.  Colonel 
Wyman  states  tnat  to  accomplisn  this  goal  first  it  is  necessary  to  define 
tne  mission.  Tne  mission  has  been  defined  as  the  Air  Defense  Mission. 
Next  tne  mission  is  broken  down  into  key  elements,  and  each  element  is 
assigned  a  quantifiaole  measure.  Tne  evaluaticn  will  examine  tne 
resultant  cnanges  in  mission  effectiveness.  Tne  center  will  also  be 
looking  at  the  maintainability,  reliability,  supportability,  and 
availaDility  of  the  JTIDS  Class  2  terminal. 

Tne  cnallenge  to  the  center  is  trying  to  equate  the  change  in  mission 
performance  with  a  cnange  in  the  pilot’s  situational  awareness.  In 
essence,  determine  tne  difference  Detween  tne  perception  and  the  reality 
of  the  pilot.  It  is  necessary  to  build  some  sort  of  process  to  assess 
reality,  wnat  is  attempting  to  be  transmitted  to  tne  pilot,  and  compare 
it  to  what  tne  pilot  is  actually  sensing.  Then  lock  at  at  the  end  of  the 
mission  and  examine  tne  discrepancies. 
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CONTROLLING  OPERATOR  WORKLOAD  IN  ARMY  SYSTEM  DESIGN 


Dr.  Lcren  Wurzman 
Army  Research  Institute 
El  Paso,  Texas 

Tms  was  a  "Government  Only  Session"  and  the  contents  will  not  De 
publisned  in  the  minutes.  Please  contact  Dr.  Wurzman  for  any  furtner 
information. 
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NEUROPSYCHOLOGICAL  WORKLOAD  BATTERY 

Mary  Thiessen 
General  Dynamics 
Fort  Wcrtn,  Texas 

M.  Tniessen  discussed  tne  Neuropsychological  Workload  Test  Battery 
Validation  Study.  This  project  is  oeing  done  in  conjunction  with  tne 
Airforce  Aerospace  Medical  Researcn  Laboratory. 

The  project  is  comprised  of  a  series  of  simulator  studies  to  assess 
pilot  workload  in  a  variety  of  air  force  environments.  Workload  is 
measured  by  recording  and  analyzing  various  pnysiological  signals  from 
tne  pilot  winile  he  performs  specific  tasks.  The  gcal  of  tne  effort  is  to 
establish  the  feasibility,  validity,  and  reliability  of  specific 
neuropnysiological  measures  contained  in  tne  test  battery  that  is 
currently  being  implemented. 

To  date  five  studies  nave  been  completed.  Final  results  on  these 
studies  will  oe  out  in  1986,  and  presently  only  preliminary  results 
are  availaole.  The  initial  test  cattery  was  designed  by  the  department 
of  psycncphysio'logy  at  tne  University  of  Illinois.  An  LSI  11  based 
system  is  responsible  for  data  acquisition,  experimental  control,  and 
some  of  the  data  analysis. 

The  original  test  battery  was  intended  to  be  a  field-usaole  device, 
out  was  really  too  cumbersome  for  that  purpose,  and  a  second  generation 
test  battery  was  built  by  Systems  Research  Laboratories  at  Dayton,  Ohio. 
This  system  is  an  LSI  II  73  based  system  and  is  much  mere  compact. 

Tne  test  cattery  measures  included  in  botn  batteries  were: 

o  Six  Brain  Measures: 


i. 

Auditory  evoked  response 

2. 

Visual  evoked  response 

3. 

Sternberg  paradigm  with  evoked  response 

4. 

Hign-Frequency  steady  state 

5. 

Steady  state  evoked  response 

6. 

Brainstem 

o  One 

Muscle  potential  measure: 

1. 

Electromycgrapnic  measure  (EMG) 

o  One 

Eye 

parameter  measure: 

1. 

Elec trcoculograpnic  measure  (EOG) 

o  One  Elec trocardiograpnic  measure  (ECG): 

1.  Heart  rate/heart  rate  vanaoility  measure. 
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Tne  first  two,  the  auditory  and  visual  evoked  responses,  were 
implemented  as  a  secondary  task  intc  the  cockpit  environment.  Actual 
data  on  tne  visual  rare  event  battery  item  sncwed  clear  results  for  the 
P300  component.  The  Sternberg  test  battery  measure  comoines  tne 
traditional  paradigm  with  tne  P300  evoked  response.  This  measure  nasn’t 
been  used  because  of  the  difficulty  implementing  it  in  the  pilot 
environment.  The  high-f requency  steady  state  measure  was  a  pre  and  post 
measure  used  to  dertermine  fatigue  effects  by  assessing  visual 
transmission  speed.  The  steady  state  evoked  response  was  implemented 
witn  the  suoject  viewing  a  monitor  with  a  grid  pattern  on  it.  The 
Drainstem  measure  was  also  only  a  pre  and  pest  measure  and  assessed 
auditory  sensitivity. 

Tne  EMG  was  intended  not  tc  measure  workload  directly,  Dut  to  get  a 
muscle  fatigue  measure.  Tne  EOG  was  recorded  by  placing  electrodes 
around  tne  subjects'  eye.  Seme  of  tne  EOG  parameters  examined  were  rate, 
amplitude  (net  too  sensitive),  50$  descent  time,  and  50$  closure 
duration.  Tne  last  two  have  been  more  sensitive  to  workload  variations. 
Tnese  measures  tend  to  decrease  witn  increasing  workload.  The  ECG 
measured  inter-beat  intervals  wnicn  were  converted  to  heart  rate. 

The  first  two  environments  used  in  the  study  were  designed  to  be  very 
simple  and  contain  3  levels  of  workload.  Tne  first  environment 
included:  l)  a  straight  .and  level  flying  task;  2)  normal  ILS;  and  3)  an 
ILS  witn  degraded  handling  qualities.  Tne  measures  tnat  were  implemented 
were  the  audi tory-evoked  response,  electromyograpmc  measure  (EMG), 
elec trcoculcgrapnic  measure  (EOG),  brainstem  measure,  and  tne  heart-rate 
measure  (HR).  Only  tne  EOG  exnibited  a  statistically  significant 
relationsnip  to  the  workload  levels.  Heart-rate  measure  was  not  analyzed 
at  this  time. 

Tne  second  environment  was  a  B-52  Tail  Gunner  environment  where  tne 
primary  task  was  tc  fire  at  targets  on  a  radar  scope.  The  manipulations 
included  tne  number  of  targets,  frequency  of  target  appearance,  and  speed 
of  targets,  just  tc  name  a  few.  The  measures  taken  were  tne 
audi tory-evoked  response,  EMG,  EOG,  and  HR.  Again,  HR  was  net  analyzed, 
and  only  the  EOG  exnibited  a  statistically  significant  relationsnip  to  no- 
load  versus  load  conditions. 

Environments  tnree  and  four  were  Day/Night  Low-Altitude  Interdiction 
Missions  conducted  in  the  F-16  simulator  at  General  Dynamics.  The 
primary  task  was  manual  terrain  following  and/or  terrain  avoidance,  and 
some  weapon  deliveries  on  targets.  The  measures  used  were  tne  EOG,  HR 
and  mgn-f requency  steady  state  evoked  potential  (SSEP)  wnicn  was 
implemented  as  a  pre  and  post  test.  Tnis  time  tne  EOG  did  not  come  out 
statistically  significant,  but  heart  rate  was  analyzed  and  exhibited  a 
sensitivity  to  lew  versus  nign  workload  conditions. 

Environment  5  was  and  Air-to-Air  Fighter  Aircraft  Intercept  Mission 
wnere  tne  pilot  was  told  to  intercept  fignters  wnich  appeared  on  his 
display.  Workload  was  varied  by  manipulating  the  type  of  enemy  aircraft 
evasive  maneuvers,  enemy  aircraft  speed,  and  initial  intercept  geometry. 
Tne  performance  measures  were  time  to  missile  release,  part  of  missile 
release  envelope,  and  aspect  angle  of  target. 
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The  original  concept  behind  this  set  of  studies  was  to  try  to  develop 
cockpit  design  guidelines.  The  second  generation  of  this  test  battery  is 
now  being  designed  from  tne  experience  gained  using  the  first  generation 
battery  in  this  set  of  studies.  The  final  report,  coming  out  in  the 
spring,  will  contain  the  results  and  conclusions  drawn  from  tnese 
studies.  For  instance  some  of  the  measures  simply  could  not  be 
implemented  into  a  fighter  environment  because  of  the  nature  of  the 
setting  and  task  demand.  Hopefully  tne  physiological  methodologies  will 
offer  enhanced  man-machine  interfaces  so  that  a  cockpit  design 
methodology  tnat  is  sensitive  to  aircrew  workload  will  be  available. 
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NEUROPSYCHOLOGICAL  WORKLOAD 
TEST  BATTERY  VALIDATION  STUDY 


AIR  FDRCE  AERDSPACE  MEDICAL  RESEARCH  LABORATORY 
CONTRACT  NO.  F3361S  82  C  0517 
PROGRAM  MANGER:  COL.  R.  D.  Q'OGNNELL 

CONTRACTOR:  GENERAL  OYNAMICS/FORT  WORTH  OIVISION 


NEUROPSYCHOLOGICAL  WORKLOAD  TEST  BATTERY 
VALIDATION  STUDY 


DEFINITION: 

•  SIMULATOR  STUOY  TO  ASSESS  PILOT  WORKLOAO  IN  A 
VARIETY  OF  AIR  FORCE  ENVIRONMENTS 

•  WORKLOAO  IS  MEASURED  BY  RECORDING  AND  ANALYZING 
NUMEROUS  PHYSIOLOGICAL  I  UNCTIONS  OF  THE  PILOT 
WHILE  HE  PERFORMS  SPECIFIC  TASKS 


GOAL: 

•  ESTABLISH  FEASIBILITY,  RELIABILITY,  ANO  VALIOITY  OF 
THE  USE  OF  NEUROPSYCHOLOGICAL  INDICES  AS  WORKLOAD 
MEASURES,  ESPECIALLY  FOR  USE  AS  A  QUANTITATIVE 
COCKPIT  SYSTEM  DESIGN  TOOL 


SCOPE: 

•  SEVEN  SEPARATE  STUDIES  WILL  BE  CONDUCTED  DVER  A  PERIDD 
OF  39  MDNTHS;  TWD  BOMBER  STUDIES  USING  CARSWELL  AFB  FACILITIES 
ANO  FIVE  FIGHTER  STUOIES,  USING  GENERAL  DYNAMICS  FACILITIES, 
WILL  BE  CONDUCTEO 
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NEUROPSYCHOLOGICAL  WORKLOAD  TEST  BATTERY 

(AKA  PEARL  II) 


•  DESIGNED  AND  DEVELOPED  BY  THE  COGNITIVE  PSYCHOPHYSIOLOGY  LABORATORY, 
UNIVERSITY  OF  ILLINOIS  AT  URBANA  CHAMPAIGN 


•  LSI  11  -  BASED  PSYCHOPHYSIOLOGICAL  DATA  ACQUISITION/EXPERIMENTAL 
CONTROL  SYSTEM 

•  CONTAINS  NINE  DIFFERENT  BATTERY  TESTS,  INCLUDING: 

•  Auditory  Oddball  Test 

•  Visual  Oddball  Test 

•  Transient  Evoked  Response  to  Visual  &  Short-Term  Memory  Tasks 

•  Brain  Stem  Response 

•  Steady-State  Evoked  Response  for  Threshold  Detection 

•  High  Frequency  Steady  State  Evoked  Response  for  Measuring 
Neural  Transmission  Speed 

•  Electromyographic  Analysis 

•  Eye-Blink/Eye  Movement  Analysis 

•  Measures  of  Heart  Rate  Variability 
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TEST  BATTERY  MEASURES 


•  BRAIN  (Electroencephalographic)  MEASURES 

*  AUOITORY  EVOKEO  RESPONSE 
'  VISUAL  EVOKED  RESPONSE 
'  STERNBERG  PARADIGM  W/EVDKED  RESPONSE 
'  HIGH-FREQUENCY  STEADY  STATE  (Neural  Transmission  Time) 
'STEADY  STATE  EVDKED  RESPONSE  (Contrast  Threshold) 
'BRAINSTEM 

•  MUSCLE  POTENTIAL  MEASURE 

'  ELECTROMYOGRAPHIC  MEASURE 

•  EYE  PARAMETER  MEASURE 

'  ELECTROOCULOGRAPHIC  MEASURE 

•  ELECTROCARDIOGRAPHIC  MEASURE 

'  HEART  RATE/HEART  RATE  VARIABILITY  MEASURE 


ENVIRONMENT  1  -  B  52  ILS  APPROACH 


PRIMARY  TASK: 

•  PHOT  PERFORMING  ILS  APPROACH 


PARAMETERS  AVAILABLE  TO  VARY  WORKLOAD: 


•  HANOLING  QUALITIES  OF  AIRCRAFT 

*  Entries  Out  -  One  Side 

*  Split  Flaps 

*  Runaway  Trim 

*  Aircraft  Loading 


PERFORMANCE  MEASURES: 


•  RMS  ELEVATION  ERROR  FROM  GLIOEPATH 

•  RMS  AZIMUTH  ERROR  FROM  GLIOEPATH 


•  CROSSWINDS 
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STUDY  1  SUMMARY 


TEST  BATTERY  MEASURES* 

RESULT 

•  AUOITORY-EVOKEO  RESPONSE 

•PATTERN  SEEN,  BUT  ANALYSIS  ALGORITHM 
SUSPECT 

•  ELECTROMYOGRAPHIC  MEASURE  (EMG) 

•  NO  SENSITIVITIES 

^ELECTROOCULOGRAPHIC  MEASURE  (EOG)** 

•  SENSITIVE  TO  LOAO  w  NO  LOAO,  STATISTI 
CALLY  SIGNIFICANT 

•  BRAINSTEM  MEASURE 

•  NO  SENSITIVITIES 

•  HEART-RATE  MEASURE  (HR) 

•  NO  SOFTWARE  AVAILABLE 

m Original  Instrumentation 
** Subcontractor  S/W 


ENVIRONMENT  2 -B-52  TAIL  GUNNER 


PRIMARY  TASK: 


•  TAILGUNNER  ACOUIRES 
TARGET  ON  RAOAR  SCOPE; 
SLEWS  CURSOR  TO  TARGET; 
FIRE 


PERFORMANCE  MEASURES: 
•  INSTRUCTOR  EVALUATION 


PARAMETERS  AVAILABLE  TO  MANIPULATE  WORKLOAD 

•  NUMBER  OF  TARGETS 

•  FREQUENCY  OF  TARGET  APPEARANCE 

•  SPEEO  OF  TARGETS 

•  AUTO/MANUAL  MOOES  OF  OPERATION 

•  RELATIVE  VECTOR  OF  TARGETS 

•  TARGET  OISCRIMIN ABILITY /DIFFICULTY  IN  PRIORITIZATION 


i 

\ 

i 
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STUDY  2  SUMMARY 


TEST  BATTERY  MEASURES* 

RESULT 

•AUDITORY  EVOKED  RESPONSE 

•  NOT  STATISTICALLY  SIGNIFICANT 

•  EMG 

•NO  SENSITIVITIES 

|eog** 

•  DISCRIMINATED  LOAO  vs.  NO  LOAD; 
STATISTICALLY  SIGNIFICANT 

•  HR 

•NO  ANALYSIS  SOFTWARE 

*  Original  Instrumentation 
#  * Subcontractor  SAV 


ENVIRONMENTS  3,4 -  DAY/NIGHT  LOW-ALTITUDE 

INTERDICTION  MISSION 


PRIMARY  TASK: 

•  MANUAL  TERRAIN  FOLLOWING 
ANO/OR  TERRAIN  AVOIDANCE 


PERFORMANCE  MEASURES: 

•  RMS  ERROR  FROM  IDEAL  ALTITUDE  PROFILE 

•  HEAOING  RMS  ERROR 

•  GROUND  IMPACTS 

•  EXCURSIONS  OUTSIDE  ACCEPTABLE  AGL  BAND 


PARAMETERS  AVAILABLE  TO  VARY 
WORKLOAD: 


•  TERRAIN  (Flit  to  Very  Mountainous) 

•  VISIBILITY  (Daylight,  Fog/Haza,  Night) 

•  AIRCRAFT  SPEEO 

•  AIRCRAFT  TF  ALTITUDE 

•  AIRCRAFT  HANDLING  QUALITIES  (Loading) 

•  RIOE  PARAMETERS  (Soft  to  Hard) 


FLIGHT  PATH  MARKER 


AZIMUTH  STEERING 
TF  STEERING  BOX 


CHAN  f  II 


HE  AVI1Y  LOADED 


soft  -  want*  aol 


I67M] 
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STUDIES  3/4  SUMMARY 


TEST  BATTERY  MEASURE* 

RESULT 

•  EOG** 

•  NOT  STATISTICALLY  SIGNIFICANT 

^  HR 

•  SENSITIVE  TO  LO  vs.  HI  WORKLOAO ; 
STATISTICALLY  SIGNIFICANT 

•  HIGH  FREQUENCY  STEADY  STATE 
EVOKED  POTENTIAL  (SSEP) 

•  NO  FATIGUE  EFFECTS 

'Replacement  Instrumentation 
“Test  Battery  SAY 


ENVIRONMENT  5-  AIR-TO-AIR  FIGHTER  AIRCRAFT  INTERCEPT 


PRIMARY  TASK: 

•  INTERCEPT  FROM  BVRTO 
VISUAL  RANGE  WITH 
AIM  9L  LAUNCHES 


PERFORMANCE  MEASURES: 


•  TIME  TO  MISSILE  RELEASE 

•  PART  OF  MISSILE  RELEASE 
ENVELOPE 

•  ASPECT  ANGLE  OF  TARGET 


PARAMETERS  AVAILABLE 
TO  VARY  WORKLOAD: 

•  ENEMY  AIRCRAFT  EVASIVE 
MANEUVERS  (Non*  to 
ThretDimtmion«l) 

•  ENEMY  AIRCRAFT  SPEED 

•  INITIAL  INTERCEPT  GEOMETRY 


m  INITIAL  HE  AO  ON  INTERCEPT. 
FORCE  ASTERN  CONVERSION 


(2)  NO  EVASIVE  MANEUVER 


0 


INITIAL  HEAO  ON  INTERCEPT 
FORCE  ASTERN  CONVERSION; 
FOLLOWTARGET  EVASIVE 
MANEUVER  UNTIL  TARGET 
IS  WITHIN  SIOEW1NOER  27* 
CONE.  FIRE  AIM  SI 


(?)  41*  ANGLE,  LEVEL 
EVASIVE  MANEUVERS 
TURN  IN  OIRECTION  A/C 
IS  APPROACHING  FROM 
TO  AVOIO  LAUNCH 
ENVELOPE 


[31  INITIAL  HEAO  ON  INTERCEPT, 
FORCE  ASTERN  CONVERSION; 
FOLLOWTARGET  EVASIVE 
MANEUVERS  UNTIL  CAN  FINE 
AIM  #t 


(?)  TARGET  PERFORMSCOMPIEX 
3  OIMENSIONAL  EVASIVE  OR 
JINKING  MANEUVERS 
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NEUROPSYCHOLOGICAL  WORKLOAD  TEST  BATTERY 
VALIDATION  STUDY 


PHYSIOLOGICAL  SIGNALS 


AMPLI 

FIL' 

IF  IE  RS. 
TERS 

DATA  PROCESSING: 

ANALYZE  PHYSIOLOGICAL  OATA  SENSITIVITY  TO  WORKLOAD 


HEART  RATE 
VARIAIIMTY 

STEAOY  STATE  IRAIN 
EVOKEO  RESPONSE 

TENSION  INOUCEO 
\  MUSCULAR  FATIGUE 

-I  wvw 

1 _  1 

<  ut 

MMC 

— 

*»«U( 

EYE  MOVEMENT 

potentials 

-I 

•  •LINK 
•SACCI 

Pv 

1  |  A  u  1 

RATE 

tOIC  VELOCITY 

COCKPIT  OESIGN  GUIDELINES  PHYSIOLOGICAL  WORKLOAO  METRICS 


•  TECHNOLOGY  INVOLVED  IN  THE  NEUROPSYCHOLOGICAL  WORKLOAD  TEST  BATTERY  OFFERS 
ENHANCED  MAN  MACHINE  INTERFACES  THROUGH  DESIGN  OF  COCKPITS  THAT  ARE  SENSITIVE 
TO  AinCREW  WORKLOAD 
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OVERVIEW  07  THE  HUMAN  PERFORMANCE  ASSESSMENT  RESEARCH  GROUP 


Sandra  G.  Hart,  Leader 

Human  Performance  Assessment  Research  Program 
NASA-Ames  Research  Center 

BACKGROUND 

The  Human  Performance  Assessment  Research  Group  was  formed  in  October  1982, 
although  isolated  work  in  this  area  had  been  performed  previously.  It  has 
focused  on  developing  generic  concepts  and  techniques  to  measure  and  predict 
workload  and  performance  and  applying  them  to  operationally  relevant  problems 
posed  by  individuals  from  other  ogranizat ions  within  NASA,  other  government 
agencies,  industry,  and  universities.  Generic  research  has  been  accomplished 
through  literature  review,  theoretical  development,  laboratory  and  simulation 
research,  and  inflight  experimentation.  Operational  problems  have  been  ad¬ 
dressed  through  technical  publications,  consultation,  and  collaborative  re¬ 
search  . 


Increasing  emphasis  has  been  placed  on  integrating  and  extending  the  results 
of  theoretical  predictions  and  empirical  research  about  human  performance  and 
cognition  with  mathematical  models  and  analytic  structures.  The  validity  and 
practical  relevance  of  the  models  and  structures  are  evaluated  in  operational 
settings  as  the  opportunities  arise.  These  activities  often  address  speci¬ 
fic  operational  problems,  although  some  are  simply  conducted  in  an  operational 
environment  to  provide  contextual  relevance. 

The  initial  goal  of  the  program  was  to  develop  measures  and  predictors  of 
human  performance  and  workload.  To  accomplish  this  objective,  a  conceptual 
framework  was  developed  in  which  the  relationships  among  task-related,  opera¬ 
tor-related,  and  performance-related  variables  were  hypothesized.  The 
underlying  assumption  was  that  performance  and  workload  are  complex  constructs 
that  each  reflect  many  variables  in  addition  to  task  requirements  and 
observable  behavior.  Although  variations  in  workload  and  performance  are  often 
directly  related,  there  are  many  instances  in  which  they  are  influenced  by 
different  factors.  Since  each  reflects  different,  important,  aspects  of  the 
successful  operation  of  a  man-machine  system,  predicting,  assessing,  and 
modifying  both  workload  and  performance,  and  their  interactions,  have  been 
studied  and  delineated.  The  typical  approach  has  been  to  to  conduct  inhouse 
research  and  to  fund  several  parallel,  theoretically  different  lines  of 
research  in  each  area  activities  with  grants  or  contract.  The  research 
efforts  at;e  coordinated  through  working  meetings  involving  all  program 
participants,  quarterly  distribution  of  research  proposals,  reports,  and 
papers  written  by  all  participants,  providing  common  experimental  paradigms 
and  tasks  against  which  different  approaches  can  be  tested,  and  interactions 
at  scientific  meetings.  More  formal  integration  of  the  disparate  research 
elements  has  been  accomplished  by  informal  program  overviews  written  each 
year,  published  reviews  of  specific  areas  (such  as  a  recent  chapter  on  the 
"Theory  and  Measurement  of  Human  Workload")  and  an  analytic  structure  that  is 
under  development  at  Ames  designed  to  predict  the  workload  and  performance  of 
different  operational  scenarios. 

Three  general  program  area  will  be  described  below.  They  are:  (l)  assessment 
techniques  for  workload  and  performance,  (2)  predictive  methods  for  workload 
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and  performance,  and  (3)  training  strategies  in  which  workload  and  performance 
assessment  are  included  as  an  integral  design  element. 

ASSESSMENT  TECHNIQUES 

The  development  of  valid  and  reliable  workload  measurement  techniques  was 
established  as  the  first  priority  of  the  program,  as  this  seemed  to  be  the 
most  pressing  requirement  of  the  aerospace  community  and  as  a  logical  precur¬ 
sor  to  other  program  elements.  Although  the  focus  was  on  workload,  appro¬ 
priate  measures  of  performance  were  developed  in  parallel,  and  comparisons 
were  made  among  the  various  measures  of  both  worklod  and  performance.  The 
measurement  areas  studied  included: 

1.  Subjective  ratings  by  operators  and  observers 

2.  Physiological  correlates 

3.  Primary  and  secondary  measures  of  precision  and  speed 

4.  Descriptive  analytic  structures 

5.  Communications  and  discourse  analyses 

In  each  area,  the  most  promising  techniques  were  selected  from  among  those 
already  in  use  and  others  were  developed.  The  candidate  techniques  were 
subjected  to  comparative  evaluations  in  laboratory  and  simulation  environments 
to  determine  their  potential  utility,  identify  unresolved  research  issues,  and 
refine  the  most  promising  ones.  In-house  research,  university  grants  and 
research  contracts  were  funded  to  allow  experts  from  a  variety  of  disciplines 
(e.g.,  psychology,  physiology,  engineering,  mathematics)  to  resolve  theoreti¬ 
cal  and  practical  problems  associated  with  the  use  of  the  selected  measures. 

As  an  example,  subjective  ratings  were  an  early  focus  of  attention  because 
the  technique  has  face  validity,  it  is  the  most  widely  and  practically  appli¬ 
cable  measure,  and  has  been  used  as  the  ultimate  validation  of  most  other 
measures.  Some  of  the  theoretical  and  practical  issues  studied  included: 

1.  Individual  differences  (a  priori  biases;  subjective  experiences) 

2.  Design  (single/multiple  dimensions;  conceptual  or  concrete  anchors; 
continuous/discrete  categories;  "fuzzy"  or  '’crisp"  descriptors) 

3.  Timing  (on-line/retrospective;  cued/free  recall) 

4.  Correlation  with  other  measures  (performance/effort/physiological 
variables /model  predictions ) 

Empirical  information  was  obtained  about  each  issue  and  practical  solutions 
were  devloped  and  tested.  An  example  of  one  such  solution  to  the  problem  of 
individual  differences  is  the  following: 

PROBLEM:  Large  individual  differences  in  workload  ratings  are  obtained 

from  individuals  who  have  experienced  apparently  similar  situations. 

HYPOTHESIS:  Differences  in  a  priori  definitions  of  "workload"  (e.g., 
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"effort",  "task  difficulty",  "success/ failure" ,  "stress",  etc)  cause  different 
raters  to  react  to  and  evaluate  different  aspects  of  a  task. 

SOLUTION *  Commonly  held  definitions  of  "workload"  were  identified.  A 
technique  was  developed  to  identify  the  relative  importance  of  each  one  (e.g. 
its  weight)  to  a  specific  operator.  These  weights  determined  the  relative 
contribution  of  each  workload-related  dimension  to  a  mathematically  derived 
workload  rating. 

VALIDATION*  Twenty  five  laboratory,  simulation,  and  inflight  experiments 
were  conducted  using  this  technique.  The  derived  workload  rating  was  compared 
to  traditional  subjective  workload  measures  (e.g.,  Cooper-Harper  Rating  Scale 
for  Aircraft  Handling  Qualities,  the  Subjective  Workload  Assessment  Technique 
developed  by  the  USAF,  and  unidimensional  ratings),  as  well  as  to  other  mea¬ 
sures  of  workload  and  performance  (e.g.,  reaction  time,  response  precision, 
heart  rate,  heart  rate  variability,  blood  pressure  variation,  evoked  cortical 
potentials,  secondary  task  performance,  etc).  In  every  experiment,  between- 
rater  variability  was  reduced  by  at  least  20%  (and  often  by  as  much  as  50%) 
and  the  percent  of  variance  accounted  for  by  the  experimental  variables  was 
increased . 

APPLICATIONS:  For  this  assessment  technique,  as  for  the  others,  the  next 
step  is  application  to  operational  problems  and  publication  in  the  open  liter¬ 
ature.  Even  though,  it  is  assumed  that  validated  measures  will  be  suffici¬ 
ently  general  that  the  information  they  provide  will  be  consistent  when  they 
are  used  appropriately,  a  subject-matter  expert  might  be  required  to  insure 
that  the  measure  or  analytic  technique  is  applied  appropriately  and  that 
reasonable  conclusions  are  drawn  when  measures  are  applied  to  specific  vehi¬ 
cles  or  operational  tasks.  Thus,  virtually  all  operational  validations  have 
been  performed  in  collaboration  with  outside  organizations. 

Again  using  the  derived  workload  rating  as  an  example,  researchers  at  Johnson 
Space  Center  requested  support  in  conducting  a  1-g  evaluation  of  two  alterna¬ 
tive  space  suit  designs  with  respect  to  upper  body  mobility  and  fatigue. 
A  battery  of  laboratory  tasks  developed  at  Ames  were  provided  as  a  focal  point 
for  the  behavorial  portion  of  the  suit-testing  research,  and  a  computerized 
version  of  the  subjective  evaluation  technique  was  provided.  This  interaction 
not  only  allowed  researchers  at  Johnson  to  take  advantage  of  the  wealth  of 
experience  already  obtained  at  Ames  with  the  tasks  and  the  measures,  but  also 
allowed  Ames  researchers  to  apply  these  measures  in  an  interesting  and  unusal 
environment.  The  logical  next  step  would  be  to  follow  the  introduction  of  the 
final  suit  design  into  space,  obtaining  performance  and  workload  measures 
under  a  variety  of  circumstances  to  evaluate  the  impact  of  the  suit  design  in 
different  environments  on  the  astronauts.  This  study  has  been  proposed  for  an 
upcoming  Shuttle  flight  in  collaboration  with  MIT. 

Finally,  a  chapter  is  in  progress  for  Human  Mental  Workload  and  an  AGARDoGRAPH 
in  which  all  of  the  applications  of  the  rating  technique  will  be  summarized  to 
facilitate  wide  dissemination  of  the  method  and  its  applications. 

PREDICTIVE  METHODS 

In  laboratory,  simulation,  and  operational  settings,  a  standardized  way  to 
create  operationally  relevant  tasks  that  impose  known,  quantified  levels  of 
workload.  is  needed  In  research  environments,  this  capability  is  needed  to 
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create  controlled,  predictable,  experimental  situations  against  which  candi¬ 
date  measurement  techniques  or  model  predictions  can  be  evaluated  or  alterna¬ 
tive  equipment  designs  tested.  In  applied  settings,  the  need  to  predict  the 
workload  of  advanced  systems  is  particularly  critical  during  the  design  phase. 
Information  about  whether  or  not  a  "typical"  operator  or  team  of  operators 
will  be  able  to  perform  the  proposed  tasks  with  acceptable  levels  of  workload 
(given  the  resources  that  will  be  available)  is  often  a  key  issue. 

Many  models  of  human  operators  exist,  however,  most  are  based  on  a  task  or 
time-line  analysis  of  the  elemental  activities  to  be  performed.  Workload 
levels  are  typically  defined  as  the  ratio  between  the  time  required  to  perform 
tasks  and  the  time  available.  Since  the  subjective  experience  of  workload  may 
involve  a  wealth  of  factors  in  addition  to  the  time  pressure  element,  and  task 
integration  and  time  sharing  are  certainly  possible  with  experience,  addi¬ 
tional  factors  must  be  included  to  derive  a  more  accurate  workload  prediction. 
In  addition,  the  detailed  nature  of  most  time-line  and  task  analytic  appro¬ 
aches  is  not  easily  extended  to  alternative  systems  or  different  levels  of 
training.  Finally,  traditional  techniques  typically  evaluate  a  new  system  in 
comparison  to  a  simular  existing  system.  The  same  procedures  may  be  difficult 
to  apply  to  entirely  novel  systems,  that  will  operate  in  unusal  environments, 
or  perform  unique  missions. 

Some  of  the  predictive  models  under  investigation  include: 

1.  Petri  net  structures  for  workload  prediction  (UCLA  -  John  Lyman) 

2.  Application  of  Fuzzy  Set  mathematics  to  the  analysis  and 

prediction  of  workload  and  performance  (U.  of  Toronto  -  Moray) 

3.  Stochastic  models  of  workload  and  performance  (Purdue  -  Townsend) 

4.  Supervisory  control  models  of  operators  of  complex  systems  (MIT 

Sheridan) 

5.  Computer-based  models  to  predict  operator  workload,  performance,  and 

task  duration  (Inhouse  -  Hart/Shively ) 

In  some  cases,  the  modeling  efforts  have  been  based  on  similar  theoretical 
models  of  workload  (e.g.,  Rasmussen's  division  of  activities  in  to  skill- 
based,  rule-based,  and  knowledge-based  tasks  that  impose  increasing  levels  of 
workload)  or  have  used  the  same  experimental  task  (e.g,  the  "POPCORN"  supervi¬ 
sory  control  task  developed  at  Ames).  The  benefit  of  these  overlapping  ef¬ 
forts  is  obyious;  comparisons  can  be  made  between  measures  and  model  predic¬ 
tions  and  information  about  the  underlying  processes  can  be  derived  from 
different  perspectives . 

A  preliminary  version  of  a  flight-task  workload  and  performance  prediction 
model  has  been  developed  at  Ames.  It  shows  great  promise  in  the  prediction  of 
workload  and  performance  levels  of  general  aviation  flights  conducted  under 
instrument  flight  rules.  It  will  be  described  below  as  one  example  of  the 
research  in  progress  in  the  area  of  prediction: 

PROBLEM:  There  is  an  urgent  need  for  a  standardized  technique  to  create 

simulation  scenarios  that  will  impose  a  predictable,  independently  determined 
level  of  workload  on  pilots  or  crews  of  pilots.  Such  scenarios  are  needed  for 
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both  research  (e.g.,  for  validation  of  candidate  measures  of  workload  and 
performance)  and  operational  environments  (e.g,,  as  a  baseline  to  compare 
alternative  instrument  designs  or  procedures). 

HYPOTHESIS*  The  workload,  duration,  and  performance  of  most  well- 
learned  tasks  is  remembered,  experienced,  and  measured  most  effectively  in 
terms  of  functionally  integral  units  (e.g.,  the  landing  segment  of  flight). 
Variations  in  workload,  duration,  or  performance  may  be  created  by  imposing 
additional  activities  on  operator(s),  modifying  the  environment,  or  changing 
the  time  constraints  within  which  the  task  is  to  be  performed.  In  addition, 
the  actual  workload  experienced  by  an  operator  during  the  performance  a 
specific  scenario  can  be  analyzed,  post  hoc,  by  computing  the  number  and  types 
of  additional  tasks  or  states  that  occurred  coincidentally  during  the  flight 
relative  to  the  intended  levels. 

SOLUTION:  A  computer  program  is  being  developed  in  which  the  functional 
units  of  instrument  flight  are  identified  and  the  nominal  durations,  perfor¬ 
mance  and  workload  levels,  and  the  transitions  from  one  to  another  are  deter¬ 
mined.  This  program  accesses  a  data  base  in  which  the  logical  variations  of 
events  or  states  that  might  occur  are  enumerated.  Finally,  the  algorithms  by 
which  different  tasks  or  states  can  be  combined  with  each  nominal  segment  have 
been  derived  empirically.  In  order  to  create  a  simulation  scenario  with  this 
model,  the  length  of  the  desired  scenario  and  the  number  of  take-offs  and 
landings  will  be  entered  by  the  user.  The  program  computes  a  nominal  scenario 
from  this  information  which  can  be  accepted  or  modified.  For  each  segment,  a 
menu  of  logical  events  that  can  be  added  or  enrivonmental  variations  that  can 
be  imposed  are  presented.  As  the  user  selects  different  options,  the  predic¬ 
ted  levels  of  workload,  performance  and  duration  are  modified  and  displayed 
graphically  and  numerically  until  the  user  has  created  a  scenario  with  the 
desired  characteristics . 

VALIDATION:  A  paper-and-pencil  version  of  the  model  has  been  tested  with 
a  large  data-base  of  workload  and  performance  levels  for  routine  flight  seg¬ 
ments  and  flight-related  events.  These  have  been  combined  to  create  candidate 
flight  scenarios  which  have  been  flown  in  a  GAT-I  simulator.  The  model  pre¬ 
dictions  have  been  tested  with  a  number  of  subjective,  secondary-task,  and 
performance-based  measures  of  workload  in  four  simulation  studies.  In  most 
cases,  the  empirical  results  support  the  the  model's  predictions. 

APPLICATIONS:  In  the  near  term,  the  paper-and-pencil  version  of  the 
model  will  be  programmed  to  allow  simulation  scenario  creation  by  computer. 
The  initial  test  of  the  computerized  model  will  be  accomplished  with  a  simula¬ 
tion  in  the  GAT-I  simulator  to  be  consistent  with  earlier  work.  The  next 
application  planned  is  for  helicopter  nap-of- the-earth  flight  with  military 
missions  projected  for  the  1990s.  The  focus  will  be  on  the  LHX  helicopter 
proposed  for  Army  use  in  the  1990s.  It  will  address  such  issues  as  one- 
versus  two- crewmember  operations.  The  model  validation  studies  for  the 
helicopter  application  will  be  conducted  at  Ames  in  the  Vertical  Motion 
Simulator  and  in  conjunction  with  the  Army  Simulation  Working  Advisory  Group. 

Additional  applications  will  focus  on  the  space  station,  with  early  develop¬ 
ment  efforts  taking  advantage  of  similar  activities  performed  in  shuttle 
operations.  A  predictive  model  of  workload  and  performance  for  the  RMS  (Re¬ 
mote  Manipulator  System)  will  be  programmed  using  the  model  structure  de¬ 
veloped  for  aircraft  and  helicopters.  It  is  anticipated  that  the  structure 
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and  algorithms  will  be  similar  across  applications,  although  the  data  bases 
(e .  g,  nominal  tasks,  transiitons,  events,  environments,  and  durations)  will  be 
task-specific.  For  the  RMS  research,  the  data  base  will  be  provided  by 
Johnson  personnel,  and  preliminary  testing  will  be  accomplished  at  Ames  in  the 
space  station  mock-up.  Final  testing  will  take  place  at  Johnson  in  the  RMS 
simulator. 


TRAINING  STRATEGIES 

For  the  purposes  of  this  program,  fulfilling  the  requirements  of  any  mission 
is  characterized  in  terms  of  different  "costs": 

1.  System  resources  required 

a.  Hardware 

b.  Software 

c.  Personnel 

2.  Training  required 

a.  Initial  system  familiarization 

b.  Recurrent  training 

c.  Transition  training  between  versions  or  positions 

3.  Operator  workload 

a.  Stress 

b.  Fatigue 

c.  Job  satisfaction 

d.  Reserve  capacity  for  emergencies,  additional  tasks 

4.  System  performance 

a.  Time  to  complete  tasks 

b.  Precision 

c.  Safety  margin 

It  is  assumed  that  these  "costs"  are  related,  that  one  may  be  traded  off  for 
another  and  that  relationships  among  them  vary  across  elements  of  an  overall 
mission.  For  example,  if  sufficient  training  is  provided,  a  task  may  be 
accomplished  with  acceptable  performance  and  workload  levels,  few  operators, 
and  unsophisticated  equipment.  If  training  is  limited  or  crew  size  reduced, 
however,  then  higher  workload  or  reduced  performance  might  have  to  be  accepted 
unless  additional  resources  are  provided  (e.g.  computer-based  automation). 

Training  is,  typically  accomplished  to:  (1)  familiarize  an  operator  with  a  new 
or  modified  system;  (2)  enhance  performance;  or  (3)  reduce  unacceptable  work¬ 
load  levels.  The  relationships  among  training,  workload,  and  performance  are 
assumed,  but  few  research  efforts  have  focused  on  the  nature  of  this  inter¬ 
relationship  nor  their  joint  measurement.  The  development  of  training  re¬ 
quirements,  strategies,  and  programs  should  be  integral  to  the  development  of 
any  new  system.  For  example,  if  the  training  required  to  achieve  acceptable 
levels  of  performance  is  prohibitive  for  a  proposed  conf inguration,  then 
alternative  solutions  such  as  computer  aiding,  additional  crew  members,  or 
revised  hardware  or  procedures  might  be  considered  early  in  the  design  pro¬ 
cess  . 
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To  design  an  appropriate  training  method,  many  issues  must  be  addressed: 

1.  Where  to  focus  training  time  and  money 

2.  Operator  qualifications 

3.  Similarity  of  proposed  system  to  existing  systems 

4.  Part-task  versus  whole-task  training 

5.  Re-integration  of  trained  part-tasks 

6.  Optimal  workload  during  training  to  maximize  effectiveness 

Whenever  a  task  is  performed,  specific  abilities  as  well  as  a  general  know¬ 
ledge  base  are  required.  Operators  experienced  on  similar  systems  or  tasks 
may  require  less  training  because  they  can  use  established  patterns  of  re¬ 
sponses,  adopt  more  efficient  strategies,  and  exert  appropriate  effort  in  a 
timely  way,  thereby  minimizing  workload.  In  addition  they  are  more  likely  to 
perceive  task  goals  and  performance  criteria  correctly.  It  is  difficult  to 

predict  skilled  performance  on  a  task  from  its  initial  performance;  general 
learning  skills  may  be  different  than  those  required  for  skilled  performance 
on  a  specific  task  and  skills  develop  at  different  rates  for  different  opera¬ 
tors  and  subtasks. 

An  extensive  literature  has  developed  in  the  last  decade  around  the  question 
of  what  attributes  of  practice  are  essential  for  improving  performance.  In 
general,  the  conclusion  has  been  that  practice  must  emphasize  consistent 
repetition  of  tasks  having  manageable  levels  of  difficulty.  To  magnify  the 
consistency  and  insure  that  training  elements  are  manageable,  complex  tasks 
are  traditionally  decomposed  into  smaller  subtasks.  The  intention  of  subtask 
training  is  to  develop  performance  "automat icity"  which  is  characterized  by 
improved  performance  quality  and  stability.  As  performance  of  each  subtask 
reaches  an  acceptable  level,  it  is  re-combined  into  the  original  complex  task. 
An  important  question  that  remains  unanswered  is  whether  or  not  a  standardized 
criterion  can  be  used  to  specify  subtask  elements  for  training.  To  date,  this 
process  has  been  more  art  than  science. 

Ordinary  experiences  suggest  that  task  demands  and  required  effort  seem  to 
decrease  with  practice.  Subjectively,  it  is  common  to  expect  that  practice 
will  reduce  the  experience  of  workload.  For  example,  novice  drivers  concen¬ 
trate  on  every  aspect  of  control:  where  to  look,  how  much  control  to  apply, 

when  to  shift,  etc.  Their  workload  is  high,  the  safety  margin  is  poor,  and 
their  ability  to  attend  to  additional  information  or  unexpected  situations  is 
limited.  In  contrast,  experienced  drivers  are  unaware  of  most  elemental 
driving  tasks;  they  perform  them  "automatically.  Subjectively,  this  is  char¬ 
acterized  by  a  dramatic  reduction  in  conscious  decision-making  and  control, 
and,  therefore,  in  workload.  Performance  quality  and  safety  margin  are  high 
and  additional  information  or  unexpected  events  can  be  handled  readily. 

Training  enables  parallel  processing,  development  of  "automatic"  behaviors, 
coordinating  previously  unrelated  activities,  and  a  reduction  in  conscious 
decision  making.  Experts  tend  to  think  in  terms  of  larger  units  of  activities 
to  avoid  being  buried  in  minutiae.  They  complete  subtasks  with  minimal  atten¬ 
tion,  recognize  patterns  of  information  and  initiate  sequences  of  actions  with 
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single  decisions.  They  develop  mental  models  of  the  system  that  allow  them  to 
predict  future  states  from  present  evidence  or  past  experience,  reducing  the 
need  to  monitor  system  status  on  a  moment- to-moment  basis.  Finally,  the 
occurrence  of  unexpected  events  or  failures  during  system  operation  may 
affect  skilled  operators  differently  than  unskilled.  They  may  disrupt  sequen¬ 
ces  of  automatic  behaviors  for  skilled  operators  (thereby  increasing  work¬ 
load).  Alternatively,  skilled  operators  can  more  readily  recognize  and  re¬ 
cover  from  their  own  errors  and  system  failures.  All  of  this  suggests  that  the 
workload  of  experienced  operators  may  be  substantially  different  than  that  of 
novices  and  that  one  measure  of  the  success  of  a  training  program  might  be  to 
achieve  target  worklod  levels  as  well  as  reaching  established  performance 
criteria.  Obtaining  objective  evidence  for  and  measurement  of  these  rela¬ 
tionship  is  necessary  to  design  training  programs  for  new  and  existing  systems 
and  to  identify  where  additional  training  might  be  beneficial. 

Thus,  in  both  the  workload  and  performance  domains,  a  phenonenon  that  can  be 
termed  "automaticity"  has  been  suggested  as  one  explanation  for  changes  that 
accompany  practice.  However,  it  has  not  been  demonstrated  that  performance 
and  workload  reflect  training  program  strategy  manipulations  similarly.  In 
addition,  the  levels  of  workload  imposed  during  training  may  influence  the 
effectiveness  of  a  training  program.  Traditionally,  workload  assessment  has 
been  used  to  compare  display  and  control  alternatives  and  to  certify  complex 
systems  for  operational  readiness.  This  research,  however,  is  intended  to  ex¬ 
plore  the  utility  of  workload  assessment  in  a  new  domain:  training  design. 
If  successful,  the  increase  in  training  efficiency  chould  be  substantial. 

One  example  of  preliminary  research  that  has  been  conducted  in  the  area  of  is 
that  conducted  by  Dr.  Vidulich  at  Ames,  described  below: 

PROBLEM:  Training  is  essential  to  insure  acceptable  levels  of  both 
performance  and  workload  in  an  operational  environment.  Yet,  no  systematic 
empirical  investigation  into  the  interaction  between  operator  workload  and 
level  of  training  has  been  undertaken. 

HYPOTHESIS:  Both  performance  improvement  and  workload  reduction  resul¬ 
ting  from  a  training  program  should  be  proportional  to  the  degree  of  cognitive 
automaticity  obtained  on  the  target  task. 

VALIDATION:  A  classification  -  task  was  presented  with  an  extended 
training  program  followed  by  a  transfer-of- training  study  from  an  auditory  to 
a  visual  presentation  mode.  The  expected  performance  improvement  occurred  and 
it  appeared  that  subjective  ratings  of  workload  reflected  an  improvement  with 
training  and  upon  transfer  to  a  different  input  modality. 

A  second  experiment  will  manipulate  task  parameters  so  as  to  influence  the 
amount  of  performance  improvement  achievable  through  practice,  replicating 
manipulations  valided  in  earlier  research.  The  goal  will  be  to  determine 
whether  subjective  ratings  of  workload  are  sensitive  to  the  training-related 
task  manipulations  and  covary  with  measures  of  performance. 

APPLICATIONS:  A  simulation  of  an  orbital  maneuvering  task  will  be  uti¬ 
lized  as  an  example  of  a  complex,  operational  task.  It  is  sufficiently  deman¬ 
ding  to  require  subtask  training.  The  size  and  duration  of  each  training 
element  will  be  defined  by  either  a  traditional  task  analysis  decomposition 
and  performance  appraisal  or  by  creating  challenging  yet  manageable  levels  of 
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workload  with  different  9ubtasks  or  groups  of  subtasks.  In  thi9  way,  the 
value  of  workload  assessment  for  training  program  design  can  be  evaluated. 

In  order  to  contribute  to  an  improvement  in  training  strategies,  this  program 
will  devote  a  substantial  part  of  its  efforts  to  the  development  of  a  inter¬ 
active,  on-line  technique  for  measuring  the  effectiveness  of  training  pro¬ 
grams,  variation  in  different  aspects  of  performance,  and  operator  workload. 
The  optimal  way  to  accomplish  this  goal  is  to  create  a  university  "center  of 
excellence”  where  research  is  ongoing  in  man/machine  integration,  automation, 
workload  assessment,  and  training.  Parallel  research  would  be  conducted  at 
Ames  using  theoretical,  empirical,  and  part-task  simulation  techniques.  Vali¬ 
dation  studies  would  be  accomplished  through  field  studies  focused  on  a  varie¬ 
ty  of  applications.  If  expertise  in  all  of  these  areas  could  .be  integrated 
and  focused  on  this  research  area,  rapid  progress  may  be  expected  in  resolving 
the  numerous  and  important  questions  that  remain  in  this  area. 

SUMMARY 

In  summary,  the  Human  Performance  Assessment  Research  Group  has  been  created 
to  understand,  assess,  and  predict  the  effects  of  task  demands,  the  environ¬ 
ment,  and  training  on  operator  workload  and  performance.  The  approach  has 
been  characterized  by  generic  research  to  discover  the  underlying  principles, 
the  development  of  assessment  techniques,  and  the  evolution  of  predictive 
methodologies.  Fundamental  research  has  been  coupled  with  vehicle-specific 
applications  of  generic  concepts  and  methods  to  general  aviation  and 
transport  operations,  rotorcraft  activities,  and  space  in  collaboration  with 
many  organizations.  Research  has  been  conducted  inhouse  with  empirical 
laboratory  and  simulation  studies  and  through  university  grants  and  research 
contracts.  There  has  been  a  concentrated  effort  to  promote  interaction  and 
integration  among  the  research  efforts  through  working  raetings,  exchange  of 
written  reports,  and  published  reviews  of  specific  program  areas.  The  work 
conducted  by  this  group  will  continue  to  be  relatively  fundamental  although  it 
will  address  applied  problems  posed  by  a  variety  of  outside  organizations  as 
the  opportunities  arise.  In  the  future,  it  is  expected  that  emphasis  will 
shift  from  methods  of  assessment  toward  predictive  models  and  integrating 
earlier  findings  with  other  research  areas  that  are  of  particular  interest  to 
the  aerospace  community  (e.g.f  training,  automation,  electronic  crewmember). 
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NASA-AMES  WORKLOAD  RESEARCH  PROGRAM 


PROGRAM  OBJECTIVE: 

DEVELOP  ANO  VALIOATE  TECHNIQUES  TO  PREOICT  ANO  ASSESS  THE  EFFECTS 
OF  TASK  OEMANOS.  ENVIRONMENT.  AND  TRAINING  ON  OPERATOR  BEHAVIOR. 
WORKLOAO.  ANO  PERFORMANCE. 


APPROACH: 

PERFORM  GENERIC  RESEARCH  TO  DISCOVER  UNDERLYING  PRINCIPLES.  OEVELOP 
ANO  VALIOATE  ASSESSMENT  TECHNIQUES.  AND  CREATE  PREDICTIVE  MOOELS. 

PERFORM  VEHICLE-SPECIFIC  APPLICATIONS  OF  GENERIC  CONCEPTS  ANO  METHOOS 
TO  ADDRESS  OPERATIONAL  PROBLEMS. 


LABORATORY  RESEARCH 


GENERAL  AVIATION  TRAINER 


SINGLE¬ 
PILOT 
.MOTION  CA! 


yc 


APPLE 


PERFORMANCE  ASSESSMENT  RESEARCH  PROGRAM 
FACILITIES 


MULTI-COCKPIT  FACILITY 
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CHITEHION 
AT  AMES: 


TASKS  DEVELOPED 


o  FITTSBERG 
o  POPCORN 


0  MULTI-COCKPIT 
SIMULATION 


0 


STANDARD  FLIGHT 
SCENARIO  MODEL 
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SCENARIO  DESIGN  PROCEDURE 


ADDITIONAL  EVENTS 


OUTPUT  OF  MODEL:  PREDICTED  WORKLOAD.  DURATION 
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MEASUREMENT 
TECHN I QUES 


MODELS 

EXPERT  OPINIONS 
PART  TASK  SIMULATIONS 


SSESSMENT 

SUBJECTIVE  RATINGS 
OBSERVER  RATINGS 
SECONDARY  TASKS 
PHYSIOLOGICAL  RECORDINGS 
PRIMARY  TASK  PERFORMANCE 
TASK  ANALYSIS 


COMMUNICATIONS  ANALYSIS:  MEASURES  OF  CREW  COORDINATION 

AND  DECISION  MAKING 


OBJECTIVE: 

ANALYZE  FLIGHT  DECK  AND  ATC  COMMUNICATIONS  TO  ASSESS  AIRCREW  DYNAMICS 
COMMUNICATIONS  COMPETENCY.  ANO  AIRCRAFT  MANAGEMENT 

APPROACH: 

o  CONDUCT  SIMULATIONS  IN  B-707 
SIMULATOR 

0  OBTAIN  POST-FLIGHT  EVALUATIONS  BY: 


(1) 

CREWMEMBERS 

12) 

EXPERTS  IN  LINGUISTIC 

AND  SEMANTIC  ANALYSIS 

(3) 

EXPERTS  IN  FLIGHT  SAFETY 

RESULTS: 

0  CREWS  DIFFERED  IN  COMMUNICATIONS 
COMPETENCY  AND  LEADERSHIP  ROLES 
0  CREW  COORDINATION  AFFECTED  DECISION 
MAKING  AND  AIRCRAFT  MANAGEMENT 
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-  00  NOT  INTERFER  WITH  PRIMARY  TASK 

-  00  NDT  BECOME  INTEGRATED  WITH  PRIMARY  TASK 

-  OPERATORS  CAN  ALLOCATE  THEIR  'RESOURCES*  FREELY  ANO  OPTIMALLY 

-  SECONDARY  TASK  PERFORMANCE  PEROICTS  PRIMARY  TASK  PERFORMANCE 
UNOER  HIGHER  LOAD  CONDITIONS  BETTER  THAN  PRIMARY  TASK 
PERFORMANCE  DOES 

□  ISSUES: 

-  A  SINGLE  SECONOARY  TASK  'YARDSTICK'  HAS  NOT  BEEN  FOUNO 

-  INDIVIDUAL  DIFFERENCES  IN  TIME-SHARING  ABILITIES 

-  SOME  FORMS  OF  COMPETITION  ARE  OESIRABLE:  SOME  ARE  NOT 

-  DIFFERENTIALLY  SENSITIVE  TO  OIFFERENT  PRIMARY  TASK  LOAO  LEVELS 

-  ARTIFICIAL  (BUT  THEORETICALLY  SOUNO)  VS  IMBEODED  (TASK-SPECIFIC 
BUT  WITH  FACE  VALIOITY)  TASKS 


NASA 

Amos  Rosejrch  Center 


SECONDARY  TASK  MEASURES: 
EXAMPLES 


o  MONITORING 
o  TRACKING 
o  TIME  ESTIMATION 
o  MEMORY  SCAN  (STERNBERG) 

o  MENTAL  ARITHMETIC 

o  ADAPTIVE  TECHNIQUES 
o  CHOICE  REACTION  TIME 
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NASA  PHYSIOLOGICAL  MEASURES: 

Amei  Beteerch  Center  EXAMPLES 


o  MEASURES  OF  MENTAL  AND  PERCEPTUAL 
PROCESSING 

M  EVOKED  CORTICAL  POTENTIALS 
*  EYE  POINT  OF  REGARD 


o 


MEASURES  OF  EMOTIONAL  AND  PHYSICAL 
ACTIVATION 


*  HEART  RATE  AND  VARIABILITY 

*  BLOOD  PRESSURE 

*  GALVANIC  SKIN  HESPONSE 
H  RESPIRATION  RATE 


*  MUSCLE  TENSION 

*  VOCAL  STRESS 

*  PUPIL  SIZE 


NASA 

Arnos  Research  Center 


HEARTRATE  AS  A 
WORKLOAD  METRIC 


hORKLOAD  • 
performance  A  A 

RESEARCH  WHOUPX**1' 


-  REFLECTS  STRESS.  NOT  BUSYNESS 

—  IS  MORE  SENSITIVE  TO  STRESS  THAN  SUBJECTIVE  RATINGS 


NASA  C-141 
WORKLOAD 
STUDY 


HEARTRATE: 


PILOT  RATING  OP 
OUKRALL  WORKLOAD 


12  14  1  €  t 

OCCHKHT  OP  PLIGHT 


Hart,  Hauser  &  Leater.  1984 
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f\l/\5A  PHYSIOLOGICAL  MEASURES: 
iwiwfi  AUDITORY  EVOKED  POTENTIALS 

Ames  Research  Cooler 


HORKLOAO 
performance 

RESEARCH  CROUP 


OBJECTIVE:  EVALUATE  THE  VALIOITY  ANO 

UTILITY  OF  AUDITORY  EVOKED  POTENTIALS 
COMPONENTS  (N400,  P300I  AS  MEASURES  OF 
WORKLOAD  ANO  FATIGUE 

APPROACH:  ERPo  WERE  ELICITEO  BY  PILOT 

CALL-SIGNS  PRESENTEO  AUD I  TOR  I  ALLY 
OURINC  CONCURRENT  TRACKING 

RESULTS:  A  SIGNIFICANT  NUMBER  OF 

SUBJECTS  EXHIBITED  CORRELATIONS 
BETWEEN: 

o  WORKLOAD  RATINGS  AND  P300  AMPL 
©  WORKLOAD  RATINGS  ANU  N400  AMPL 
©  FATIGUE  ANO  P300  AMPL 


*  PROVIDE  SIGNIFICANT  SOURCE  OF  INFORMATION 

*  MAY  TAP  THE  ESSENCE  OF  MENTAL  WORKLOAD 

*  REFLECT  SUBSET  OF  INFORMATION  AVAILABLE 
DURING  TASK  PERFORMANCE 

-  RESULTS  OF  INFORMATION  PROCESSING 

-  MEMORIES 

-  OVERT  BEHAVIOR 

-  FEELINGS 

*  INDIVIDUAL  DIFFERENCES  IN  DEFINITION  AND 
AND  EXPERIENCE 

*  NO  MENTAL  REFERENCE  SCALE  FOR  "WORKLOAD  * 

*  BEST  TO  COMPARE  SHARED  QUALITIES  AND 
SIMILAR  ACTIVITIES 

*  CALIBRATION  OF  RATERS 

*  TIMING 

-  ON-LINE  vb  RETROSPECTIVE 

-  PRIMACY/RECENCY  OR  ODDBALL  EFFECTS 

*  PSYCHOMETRIC  CONSIDERATIONS 

-  EQUALITY  OF  INTERVALS 

-  NO  "ZERO'  POINT  OR  "MAXIMUM" 
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DEVELOPMENT  OF  A  SUBJECTIVE  RATING  SCALE  FOR  WORKLOAD 

OBJECTIVE:  EMPIRICALLY  DETERMINE  THE  NECESSARY  TASK.  BEHAVIOR. 

PERFORMANCE.  AND  SUBJECT-RELATED  SUBSCALES.  THE  RULES  BY  WHICH 
THEY  MAY  BE  COMBINED  TO  ESTIMATE  DVERALL  WORKLOAD.  AND  THE 
IMPLEMENTATION  AND  DESIGN  DF  THE  SCALES. 

APPROACH:  DBTAIN  INFORMATION  ABDUT  THE  IMPORTANCE.  MAGNITUDE.  AND 

SENSITIVITY  OF  10  CANDIDATE  FACTORS  IN  15  EXPERIMENTS  THAT  INCLUDED 
SIMPLE  MANUAL  CONTRDL.  DECISION  MAKING  AND  INFORMATION  PROCESSING. 
SUPERVISORY  CONTROL.  AND  SIMULATED  FLIGHT. 

RESULTS: 

1.  SDURCES  AS  WELL  AS  MAGNITUDES  OF  WORKLOAD  DIFFERED  WITHIN  AND 
BETWEEN  EXPERIMENTAL  TASKS 

2.  INDIVIDUAL  WORKLDAD  DEFINITIONS  VARIED  WIDELY 

3.  SIX  FACTORS  WERE  SELECTED  AS  THE  OPTIMAL  WORKLOAD  SUBSCALES: 

MENTAL.  PHYSICAL.  S  TEMPORAL  DEMANDS.  PERFORMANCE.  EFFDRT  G  FRUSTRATION 

4.  20  SCALE  INTERVALS  LIMITED  BY  BIPOLAR  ANCHDRS  PROVIDE  A  PRACTICAL  AND 
SUFFICIENTLY  SENSITIVE  MECHANISM  TO  OBIAIN  SUBSCALE  MAGNITUDE  RATINGS 

5.  A  COMPARISON  AMONG  THE  SIX  SUBSCALES  TO  SPECIFY  THEIR  IMPORTANCE  AS 
SOURCES  OF  WORKLOAD  PRDVIDE  IMPORTANCE  WEIGHTS  FOR  EACH  SUBJECT 

G.  THE  AVERAGE  OF  THE  MAGNITUDES  RATINGS  FOR  THE  SIX  SUBSCALES.  EACH 
WEIGHTED  BY  THE  IMDRTANCE  ATTACHED  TO  THEM  AS  A  WORKLOAD-SOURCE  FDR 
THAT  TASK.  PROVIDES  A  STABLE  AND  SENSITIVE  WDRKLDAD  INDEX. 


THE  TYPES  OF  EXPERIMENTAL  TASKS  INCLUDED  IN  THE  WORKLOAD  RATING  SCALE 

DEVELOPMENT  EFFDRT 


0  SIMPLE.  COGNITIVELY-LOADING  TASKS 

CHOICE  REACTION  TIME.  MEMORY  SEARCH.  MENTAL  ARITHMETIC. 

MENTAL  ROTATION.  PATTERN  MATCH 

0  SIMPLE.  MANUALLY-LOADING  TASKS 
ONE  AND  TWO  AXIS  TRACKING 

0  CONCURRENT.  INDEPENDENT  DUAL-TASKS 

TRACKING  +  MEMORY  SEARCH.  MENTAL  ROTATION 

0  SERIAL.  INTEGRATED  'FITTSBERG*  TASKS 

TARGET  ACQUISITION  +  MEMORY  SEARCH.  MENTAL  ARITHMETIC.  RHYMING. 

PATTERN  MATCH.  PREDICTION,  TIME  ESTIMATION 

0  COMPLEX  SUPERVISORY  CDNTROL  SIMULATIONS  ('POPCORN') 

0  PART-TASK  AND  FULL-MISSION  AIRCRAFT  SIMULATIONS 
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SUBJECTIVE.  RATINGS 
METHOD  FOR  REDUCING  THE 
BETWEEN  SUBJECT  VARIABILITY 


BIPOLAR  RATINGS: 


WORKLOAO  DIMENSIONS: 


THE  AMOUNT  OF  EACH  FACTOR 
EXPERIENCED  IN  A  TASK  IS 
EVALUATED  ON  A  BIPOLAR 
SCALE: 


irirJSLid? 


ff-  wiLur. 


WEIGHTS 


EACH  OF  9  FACTORS  IS  COMPARED  WITH 
EVERY  OTHER  ONE  (WHICH  IS  MORE  RE¬ 
LATED  TO  WORKLOAO?) 


WEIGHTING  PROCEDURE 


EACH  'RATING"  IS  WEIGHTED  BY 
ITS  SUBJECTIVE  IMPORTANCE  TO 
EACH  SUBJECT  (THE  'WEIGHTS') 


MENTAL  EFFORT 


THE  AVERAGE  OF  THE  WEIGHTED 
mnHGS^_DEaiy£QJ(0RKL0A0^C0RI 


0  -  NEVER  SELECTED 


• 

•  »l«  ViniClAfV 

• 

f|N(  MMSORf 

• 

Om«  flMOtNAMEt 

• 

pmtiicm.  cvtoor 

• 

wntdi  crroof 

• 

(Mill 

• 

f  At  I CUC 

• 

routtoAt ion 

• 

TYPf  o *  l«M 

TASK  DIFFICULTY 

HI1 

< 

LO 

SUBSCALES  SELECTED  FOR  NASA  WORKLOAD  RATING  SCALE 


TASK  RELATED: 

0 

MENTAL  DEMANDS 

(MD) 

0 

PHYSICAL  DEMANDS 

(PD) 

0 

TEMPORAL  OEMANDS 

(TD) 

RESPONSE-RELATED: 

0 

EFFORT  EXPENDED 

(EF) 

0 

PERFORMANCE  QUALITY 

(OP) 

0 

FRUSTRATION  LEVEL 

(FR) 
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APPENDIX  A 


EXAMPLE:  COMPARE  WORKLOAD  OF  TWO  TASKS  THAT  REQUIRE  A  SERIES  OF 

DISCRETE  RESPONSES.  THE  PRIMARY  DIFFICULTY  MANIPULATION  IS  THE 
INTER-STIMULUS  INTERVAL  (ISI) -  (TASK  1  -  500  MSEC.  TASK  2  -  300  MSEC) 


PAIR-WISE  COMPARISONS  OF  FACTORS: 

INSTRUCTIONS:  SELECT  THE  MEMBER  OF  EACH  PAIR  THAT  PROVIDED  THE  MOST 

SIGNIFICANT  SOURCE  OF  WORKLOAD  VARIATION  IN  THESE  TASKS. 


PD 

/ 

MD 

TP 

/ 

PD 

TP 

/ 

FR 

TP 

/ 

MD 

OP 

/ 

PD 

TP 

/ 

EF 

OP 

/ 

MD 

FR 

/ 

PD 

OP 

/ 

FR 

FR 

/ 

MD 

EF 

/ 

PD 

OP 

/ 

EF 

EF 

/ 

MD 

TP 

/ 

OP 

EF 

/ 

FR 

TALLY  OF  IMPORTANCE 
SELECTIONS: 

MD  III  -  3 
PD  -  0 

tp  iiiii  -  5 

OP  I  -  1 

FR  III  -  3 

EF  III  -  3 


SUM  -  15 


RATING  SCALES: 

INSTRUCTIONS:  PLACE  A  MARK  ON  EACH  SCALE  THAT  REPRESENTS  THE  MAGNITUDE 

OF  EACH  FACTOR  IN  THE  TASK  YOU  JUST  PERFORMED. 


RATINGS  FOR  TASK  1: 


MD  LOW  I _ I 

PD  LOW  I _ I 

TD  LOW  I _ I 

OP  EXCL  I _ I 

FR  LOW  I _ I 

EF  LOW  I _ 


RATINGS  FOR  TASK  2: 


MD 

LOW 

I 

_  _  I 

PD 

LOW 

I  _  - 

_ I 

TD 

LOW 

I  _ 

_ I 

OP 

EXCL 

I 

_  I 

FR 

LOW 

I _ 

_  I 

EF 

LOW 

I 

_ I 

RATING 

WEIGHT 

PROOUCT 

HIGH 

30 

X 

3 

• 

90 

HIGH 

15 

X 

0 

- 

0 

HIGH 

60 

X 

5 

- 

150 

POOR 

•40 

X 

1 

- 

40 

HIGH 

30 

X 

3 

- 

90 

HIGH 

40 

X 

3 

- 

120 

SUM 

- 

490 

WEIGHTS (TOTAL) 

- 

15 

MEAN 

WWL 

SCORE 

- 

32 

RATING 

WEIGHT 

PROOUCT 

HIGH 

30 

X 

3 

m 

90 

HIGH 

25 

X 

0 

m 

0 

HIGH 

70 

X 

5 

m 

350 

POOR 

50 

X 

1 

m 

50 

HIGH 

50 

X 

3 

m 

150 

HIGH 

30 

X 

3 

m 

90 

SUM 

m 

730 

WEIGHTS (TOTAL) 

m 

15 

MEAN 

WWL 

SCORE 

m 

49 

RESULTS:  SUBSCALES  PINPOINT  SPECIFIC  SOURCE  OF  WORKLOAD  VARIATION 

BETWEEN  TAKS  (TP) .  THE  WWL  SCORE  REFLECTS  THE  IMPORTANCE  OF  THIS 
AND  OTHER  FACTORS  AS  WORKLOAD-DRIVERS  AND  THEIR  SUBJECTIVE  MAGNITUDE 
IN  EACH  TASK. 
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VALIDATION  OF  NASA  WORKLOAD  ASSESSMENT  MEASUREMENT  BATTERY 


OBJECTIVE:  OETERMINE  THE  SENSITIVITY  AND  OPERATIONAL  VALIOITY  OF  THE 
WORKLOAO  MEASURES  OEVELOPEO  AT  NASA-AMES 

APPROACH:  CONSTRUCT  SCENARIOS  WITH  WORKLOAO  PREOICTIVE  MOOEL 

PERFORM  FLIGHTS  IN  B-727  SIMULATOR  ANO  SH-3  HELICOPTER 
COMPARE  MOOEL  PREDICTIONS  TO  EMPIRICAL  RESULTS 

MEASURES:  PERFORMANCE  (COMMUNICATIONS.  ERRORS.  CREW  COORDINATION. 

CONTROL  VARIABILITY.  SECONOAHY  TASKS) 

PHYSIOLOGICAL  (HEART  RATE/VARIABILITY.  EYE  BLINK  RATE/TIMING. 

SCAN  PATTERN.  AUOITORY  EVOKEO  CORTICAL  POTENTIALS) 
SUBJECTIVE  (NASA  MULTI-OIMENSIONAL  SCALE.  REFERENCE  TASK 
COMPARISON.  MOOIFIEO  COOPER-HARPER  SCALE) 


COCKPIT 


MVSflr 


727 


HELICOPTER 


SH 


SELECTING  APPROPRIATE  WORKLOAO  ASSESSMENT. MEASURES 
(NASA  ’EXPERT*  SYSTEM) 


FORMALIZE  THE  FOCUS  OF 
A  WORKLOAD  QUESTION: 


MISSION  REQUIREMENTS? 

+  DESIGN  ALTERNATIVES? 

SYSTEM  PERFORMANCE? 
ASSIGNMENT  OF  DUTIES? 

(MAN  OR  MACHINE?) 

CREW  COMPLEMENT? 

TRAINING  REQUIREMENTS? 

CREW  SATISFACTION.  HEALTH7 


DETAILED  DESCRIPTIONS 
OF  CANDIDATE  MEASURES 
(REFERENCES) 


INSTRUCTIONS  FOR 
APPLICATION  ANO 
ANALYSIS 


APPROPRIATE  MEASURES 
GIVEN  THE  QUESTION  00 
AND  THE  ENVIRONMENT  (+) 


SPECIFY  RESEARCH 
ENVIRONMENT: 


SUBJECTIVE  RATINGS  f 
SECONDARY  TASK  PERFORMANCE 
PRIMARY  TASK  PERFORMANCE 
VISUAL  SCAN  PATTERN 


LABORATORY 
»  SIMULATION 
INFLIGHT 

UNDER  DEVELOPMENT 
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ATTENDANCE  LIST 
Workload  SuoTAG  Meeting 
5  November  1985 


President 

USA  Infantry  Board 
ATZB-IB-ATS-HF( Tom  Greene) 
Fort  Benmng,  GA  51905 


Mr.  Wayne  Banks 
NUSC 

Code  5251,  Bldg.  80 
New  London,  CT  06520 

Commander 
USA  CDEC 

Office  Symbol  ATEC-TP 
ATTN:  Major  Bennett 

Fort  Ord,  CA  95941 

Robert  Earing 

Pacific  Missile  Test  Center 
Cede  4025 

Point  Mugu,  CA  95042 

Mr.  David  R.  Eike 
Carlow  Associates,  Inc. 

8515  Lee  Hignway 
Fairfax,  VA  22051 

Dr.  Carl  E.  Englund 

Naval  Health  Research  Center 

P.0.  Bex  85122 

San  Diego,  CA  92158-9174 

R.  F.  Gaonel,  Pn.D. 

Manager,  Human  Factors  Engineering 
C1-E71,  Code  55-36 
Douglas  Aircraft  Co. 

3855  Lakewock  Blvd. 

Long  Beach,  CA  90846-0001 

Mr.  Jonn  F.  Garren 

Chief,  Flight  Management  Division 

MS  469 

NASA-Langley  Research  Center 
Hampton,  VA  25665-5225 

MAJOR  Lucky  A.  Goeoel 
401  Test  Squadron/TDD 
Dyess  AFB,  TX  79607-5270 

Mr.  Heroert  Goldman 

Code  4025  (Human  Factors  Brancn) 

PMTC 


Ms.  Sandra  G.  Hart 
ATTN:  MS  259-5 
NASA-Ames  Researcn  Center 
Moffett  Field,  CA  94055 

Dr.  E.  M.  Haverland 

U.S.  Army  Operational  Test  &  Eval.  Agency 

ATTN:  CSTE-ASD-T 

5600  Cclumoia  Pike 

Falls  Cnurcn,  VA  22041-5115 

Ernest  Hcrton 

USA  Yuma  Prcving  Ground 

Attn:  STEYP-MT-ES,  (E.  Horton) 

Yuaa,  A Z  85365-9103 

Mr.  Mark  Ranter 

Code  5512,  Bldg.  1171-1 

Naval  Underwater  Systems  Center 

Newport,  RI  02841-5047 

Mr.  Keitn  Karn 

Aircrew  Systems  Dept.,  code  SY  727 

Naval  Air  Test  Center 

Patuxent  River,  MD  20670-5504 

Steven  A.  Kinsley 
Code  6021 

Naval  Air  Development  Center 
Warminster,  PA  18974 

LTC  Gerald  P.  Krueger,  Ph.D. 

Walter  Reed  Army  Institute  of  Researcn 
ATTN:  SGRD-UWI-C/KRUEGER 
Washington,  DC  20507-5100 

LCDR  Dennis  McBride 
Code  4025 

Pacific  Missile  Test  Center 
Point  Magu,  CA  95042-5000 


Point  Mugu,  CA  95042  Mr.  Gordon  K.  McElroy 

Cede  5152 

Naval  Weapons  Center 

Enclosure  1  Cnina  Lake,  CA  95555 

D.  Reed  Morgan 
AFWAL/AAAS-2 

L-47  ' Wrignt-Patterson  AFB,  OH  45455 


LT  COL  Mark  Nataupsky 
MS  156A 

NASA-Langley  Researcn  Center 
Hampton,  VA  23665-5225 

Mr.  Lee  E.  Paul,  ACT-240 

FAA  Technical  Center 

Atlantic  City  Airport,  NJ  08405 

LT  COL  Joe  BoD  Phillips 

HQ  TAC/DRFA- Advanced  Tactical  Fignter 

Langley  AFB,  VA  23665 

LCDR  Micnael  J.  Pianka 

Code  SY  70P 

Naval  Air  Teat  Center 

Patuxent  River,  MD  20670-5304 

Mr.  Lockwood  Reed 

US  Army  Avionics  R&D  Activity 

SAVAA-C 

Ft.  Monmouth,  NJ  07703 

Mr.  Gary  Reid 
AFAMRL/HEG 

Wrign t-Pa tterson  AFB,  OH  45433 

Dr.  Stepnen  M.  Rckicki 
HQ  AFOTEC/OAH 

Kirtland  AFB,  NM  87117-5000 

Dr.  Hendrick  W.  Ruck 
AF  Inspection  &  Safety  Center 
HQ  AFISC/SEL 

Norton  AFB,  CA  92409-7001 

Dr.  Eduardo  Salas,  Pn.D 
Code  712,  Human  Factors  Division 
Naval  Training  Systems  Center 
Orlando,  FL  32813-7100 

Dr.  Richard  Schiffler 
ASD/ENECH 

Wngnt-Patterson  AFB,  OH  45435 

Dr.  Larry  Snadow 

6545  Test  Group/TIE 

Hill  AFB,  UT  84056-5000 

LCDR  T.  J.  Singer,  MSC,  USN 
Code  60B5 

Naval  Air  Development  Center 
Warminster,  PA  18974-5000 


Ms.  Janice  Stoli*cer 

6520  Test  Group/ENAH,  Stop  239 

Edwards  AFB,  CA  93523 

Dr.  William  F.  Storm,  Ph.D 
Crew  Tecnnology  Division 
USAFSAM/VN 

Brooks  AFB,  TX  78235-5000 

Ms.  Mary  Tniessen 
General  Dynamics 
P.0.  Box  748 
MZ  1766 

Ft.  Wortn,  TX  76101 

Dr.  Heatner  Wild 
Code  6021 

Naval  Air  Development  Center 
Warminster,  PA  18974 

Dr.  Loren  Paul  Wurzman 
ARI 

P.0.  Box  6057 
El  Paso,  TX  79916 

COL  Jim  Wyman 

Air  Force  Operational  Test  &  Eval.  Ctr. 
AFOTEC/Det  2  AJ  Coj\m  Test  Team 
Eglin  AFB,  FL  32542 
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ATTACHMENT  M.l 


USER-COMPUTER  INTERACTION 


ATTENDEE  LIST 


DEPARTMENT  OF  DEFENSE 
HUMAN  FACTORS  TECHNICAL  ADVISORY  GROUP 
USER-COMPUTER  INTERACTION  SubTAG 
ATTENDEE  LIST 
5  NovemDer  1985 


Dr  .  LI i  1  1  i  am  Askren 
AFHRL/LR 

Llr  i  gh  t-Pat  terson  AFB  ,  OH  45433 

(513)  255-3713 

785-3713 

Mr.  Wayne  Banks 
NUSC 

Code  3512,  Bldg.  1171-1 
Newport,  RI  02841-5047 
(203)  440-6192 
948-6192 

Dr.  Mark  M.  Brauer 
AIIE  Rep.,  Lockheed  Corp . 

Box  551,  Dept.  6951,  Bldg.  80,  Pit  A-l 
Burbank,  CA  91520-6951 
(818)  847-4345/1552 


Mr.  Ger al d  Cha i k i n 

HEL  Det.-MICOM,  SLCHE-Mf  '( Cha  i  k  i  n ) 

US  Army  Missile  Command 
Redstone  Arsenal ,  AL  35898-7290 
(205)  876-2048 
746-2048;  FTS  876-2048 

Dr.  Thomas  M.  Cook 

Martin  Marietta  Aerospace,  M.S.  L80 1 0 
P.O.  Box  179 
Denver,  CO  80210 
(303)  971-5698 


Mr .  Tuan  Doan 
SA-ALC/MMI RGC 

Kelly  AFB,  TX  78229 
(512)  925-7494 
945-7494 

Mr  .  Bob  Ear i ng 
PMTC 

Code  4025 

Point  Mugu,  CA  93042 
(805)  989-9408 
351-9408 


Ms .  Heidi  Fiedler 

Naval  Underwater  Systems  Center 

Code  3512,  Bldg.  1171-1 

Newport,  RI  02841-5047 

(401)  841-2648 

948-2648 

Mr .  Cl ar ence  Fry 

U.S.  Army  Lab.  Command;  HEL 

ATTN:  SLCHE-AD  (Fry) 

Aberdeen  Proving  Ground,  MD  21005-50 

(301)  278-5834 

298-5834 

Dr.  Richard  F.  Gabriel 

SSS  Rep.,  Douglas  AC.  Co.,  C1-E71 

3855  Lakewood  Blvd. 

Long  Beach,  CA  90846-0001 
(213)  593-8642 


Mr.  John  F.  Garren 
MS  469 

NASA-Langley  Research  Center 
Hampton,  VA  23665 
(804)  865-2233 
FTS  928-2233 

Mr.  Steve  A.  Heckart 

Aerospace  Medical  Research  Lab. 

AAMRL/HED 

Wr i gh t-Pa t ter son  AFB,  OH  45433-6573 

(513)  255-7593 

785-7593 

Dr.  Charles  C.  Jorgensen 
Oak  Ridge  National  Laboratory 
P.O.  Box  "X" ,  Bldg.  6025  MS-6N 
Oak  Ridge,  TN  37830 
(615)  574-6319 


Mr.  Maurice  A.  Larue,  Jr. 

Al A  Rep.,  Martin  Marietta  Corp. 
L8010,  P.O.  Box  179 
Denver ,  CO  8020 1 
(303)  977-0479 


Mr.  David  R.  Eike  Dr.  Patrick  H.  McCann 

Carlow  Associates,  Inc.  Navy  Personnel  R&D  Center 

8315  Lee  Highway  Code  71 

Fair-fax,  VA  22031  San  Diego,  CA  92152 

(703)  698-6225  (619)  225-2081 

*33-2081 

M  1-1  _ _ 


Capt.  Robert  K.  McCoy 

AFOTEC/OAHF 

Mr.  Larry  Peterson 

USAHEL,  Bldg.  459 

ATTN:  SLCHE-CC  <Larry  Peterson) 

Kirtland  AFB,  NM  87111-7001 
<505)  844-9606 

244-9606 

Aberdeen  Proving  Ground,  MD  21005-^B 
<301)  278-5957 

298-5957 

Mr.  Michael  D.  McNeese 

AFAMRL/HEC 

Dr.  E.R.N.  Robinson 
<NPRDC) 

3456  Wisteria  Dr. 

Wr i gh t-Pat terson ,  OH  45433-6573 
<513)  255-8805 

785-8805 

San  Diego,  CA  92106 
<619)  225-4395 

933-2081 

Mr.  Angelo  Mirabella 

OSD 

12508  Kuhl  Road 

Silver  Spring,  MD  20902 
<202)  695-9759 

225-9759 

Dr.  Marianne  Rudisill 

NASA- John  son  Space  Center 

Code  SP22 

Houston,  TX  77058 
<713)  483-4065 

FTS  525-4065 

Mr .  Me  1 vyn  Moy 

NPRDC 

Code  71 

San  Diego,  CA  92152 
<619)  225-2081 

933-2081 

Dr.  Richard  J.  Schiffler 

ASD/ENECH 

% 

Ur i gh t-Pat ter son  AFB,  OH  45433-6503 
<513)  255-5597 

785-6925 

Mr.  Louis  B.  Myers 

Battel  It  Columbus  Labs 

505  King  Ave . 

Columbus,  OH  43201 
<614)  424-7166 

Ms.  G.  R.  Eugenia  Schneider 

Code  3191 

Naval  Weapons  Center 

China  Lake,  CA  93555-6001 
<619)  939-5969 

437-5969 

Dr .  John  J .  O'  Hare 

Office  of  Naval  Research,  Code  1142 
800  N .  Quincy  St . 

Arlington,  *JA  22217-5000 
<202)  696-4502 

226-4502 

LCDR  Edward  Trautman 

EP  Code  31— D 

NPRDC 

San  Diego,  CA  92152 
<619)  225-7109 

933-7109 

Dr.  Glen  Osga 

Naval  Ocean  Systems  Center 

Code  441 

San  Diego,  CA  92152-5000 
<619)  225-7372 

933-7372 

Dr .  Heather  Wild 

Naval  Air  Development  Center 

Code  6021 

Warminster,  PA  18974-5000 
<215)  441-2561 

441-2561 

Mr.  Lee  E.  Paul 

ACT-240 

FAA  Technical  Center 

Atlantic  City  Airport,  NJ  08405 
<609)  484-5764 

FTS  482-5764 

Dr.  Loren  Paul  Uurzman 

ARI 

P.0.  Box  6057 

El  Paso,  TX  79906-0057 
<915)  568-4491/5297 

978-5297 
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ATTACHMENT  M.2 


STARS : 


SOFTWARE  TECHNOLOGY  FOR 


ADAPTABLE,  RELIABLE  SYSTEMS 


DODMCCR  NEEDS 


Standardize  useful  software  modules  and  tasks 
Maximize  software  transportablllty/usablllty 
Enhance  system  reliability 
Enhance  software  design  productivity 
Assess  user  population 
Develop  technology  base- 
Develop  SEE" 

integrate  emerging  technologies 


SEE 

METHODS 

MEASURES 

BUSINESS  PRACTICES 
HUMAN  RESOURCES 
APPLICATIONS 


STARS 

A  SYSTEMATIC  RESPONSE 
TO  SYSTEMIC  CRISIS 


SOFTWARE 
PAID  FOR  BUT 
NEVER  DELIVERED 
($1.95  MILLION) 


SOFTWARE  THAT 
COULD  BE  USED 
AS  DELIVERED 
($0.12  MILLION) 


SOFTWARE  THAT 
COULD  BE  USED 
AFTER  CHANGES 
($0.2  MILLION) 


SOFTWARE  USED  BUT 
EXTENSIVELY  REWORKED 


OR  LATER  ABANDONED 
($1.3  MILLION) 


RESULTS  OF  AN  AUDIT  OF  NINE  DOD  SOFTWARE 
DEVELOPMENT  EFFORTS  TOTALING  $6.8  MILLION 

(SOURCE:  COMPTROLLER  GENERAL  REPORTTO  CONGRESS) 


M.  2-1 


STARS  MISSION 


TO  ENHANCE  SOFTWARE  USABILITY,  MAINTAINABILITY  AND 
PRODUCTIVITY  BY  IMPROVING  THE  STATE  OF  PRACTICE  IN 
THE  ACQUISITION,  MANAGEMENT,  DEVELOPMENT  AND 
SUPPORT  OF  SOFTWARE  FOR  MISSION-CRITICAL  COMPUTER 
RESOURCES  (MCCRs). 


STARS  OBJECTIVES 


1 .  ENHANCE  THE  POWER  OF  TOOLS  BY: 

•  IMPROVING  PROJECT  MANAGEMENT  TOOLS 

•  IMPROVING  APPLICATION-INDEPENDENT 
TECHNICAL  TOOLS 

•  IMPROVING  APPLICATION-SPECIFIC 
TECHNICAL  TOOLS 


IMPROVED  PROJECT 
MANAGEMENT  TOOLS 


IMPROVED  APPLICATION 
DEPENDENT  TOOLS 


IMPROVED  APPLICATION 
INDEPENDENT  TOOLS 


\2^rv 


ENHANCED  TOOLS 


M.  2-2 


STARS  OBJECTIVES 


2.  IMPROVE  PERSONNEL  RESOURCES  BY: 


.  INCREASING  THE  LEVEL  OF  EXPERTISE 

.  EXPANDING  THE  BASE  OF  EXPERTISE 
AVAILABLE  TO  DoD 


INCREASED  LEYEL 
or  EXPERTISE 


EXPANDED  BASE 
OF  EXPERIENCE 


STARS  STRATEGY 

A  THREE-STAGE  EVOLUTIONARY  APPROACH: 


STAGE  1  -  CONSOLIDATION  of  demonstrated  tools, 
techniques,  practices,  educational  programs,  and  other 
technologies  into  a  state-of-the-art  programming 
environment. 


i  ■■  a 

STATE-OF-THE-ART 
PROGRAMMING  ENVIRONMENT 


M.  2-3 


STARS  STRATEGY 

A  THREE-STAGE  EVOLUTIONARY  APPROACH: 


STAGE  2  -  ENHANCEMENT  of  the  Stage  1  environment 
through  application/creation  of  emerging  technologies 


PROGRAMMING  ENVIRONMENT 


STARS  STRATEGY 

A  THREE-STAGE  EVOLUTIONARY  APPROACH: 


STAGE  3  -  TRANSITION  to  a  steadily  evolving  environment 


SEE 


THE  SOFTWARE  CRISIS 
-COST- 


1980  1982  1984  1986  1988  1990 


PROJECTED  ECS  COSTS  IF  CURRENT 
TRENDS  CONTINUE 

(SOURCE:  EIA) 

P773  SOFTWARE 
—  HARDWARE 


SEE 


THE  SOFTWARE  CRISIS 
-  QUALITY  - 


UNRESPONSIVE  -  Does  not  meet  user  needs 

UNRELIABLE  -  Does  not  ensure  error-free  performance 

COSTLY  -  Frequently  over-budget;  often  excessive 

UNMODIFIABLE  -  Maintenance  Is  complex,  costly  and 

error-prone 

UNTIMELY  -  Frequently  late;  often  less  capable  than 
advertised 

NON-PORTABLE  -  Software  from  one  system  will  not 

run  on  another  system  -  even  when 
functions  are  similar 

INEFFICIENT  -  Development  efforts  do  not  optimize 
utilization  of  resources  (processing  time, 
memory  space) 
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STARS 


THE  SEE  MISSION 


Support  of  MCCR*  software  engineering  through 
all  phases  of  the  software  life  cycle 

Direct,  computer-based  support  of  the  life  cycle 
functions  themselves,  including  support  for  the 
programming  languages  in  which  the  MCCR 
sotware  is  encoded 

Management  of  MCCR  software  products 

Support  of  the  users  who  develop,  support, 
reuse,  manage  and  control  MCCR  software 


*MCCR  =  Mission  Critical 
Computer  Resource 
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SEE  EMPHASIS 


STANDARDIZATION  of  all  interfaces,  tools, 
methods,  measures  and  business  practices 

COMPATIBILITY  across  modules 

TRANSPORTABILITY  between  systems 

INTEGRATION  of  all  human,  machine  and 
software  elements 


ITERATIVE  DESIGN 
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HUMAN  ENGINEERING  FOR  STARS 

1985  EFFORT 

TASK  1 

STARS  Human  Engineering  Program  Plan 


Objective:  To  develop  an  overall  STARS  human 
engineering  program  plan  that  Integrates  the 
various  STARS  elements 


Status:  Draft  program  plan  submitted  for 
review  and  comment 


HUMAN  ENGINEERING  FOR  STARS 

1985  EFFORT 


TASK  2 

STARS  Program  Support 


Objective:  To  support  the  STARS  program  in 
terms  of  planning  technology  development, 
technical  assessments  and  surveys,  and  human 
factors  reviews  of  program  products 


Status:  Technology  plan  in  preparation  for 
developing  user  interface  requirements  for 
STARS  products  employing  emerging  technology. 
Major  emphasis  is  on  functional  allocation  in 
automated  systems.  Scheduled  delivery  by  31 
October  1985. 
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HUMAN  ENGINEERING  FOR  STARS 

1985  EFFORT 


TASK  3 

Planning  and  Coordination  of  Symposia  and 

Workshops 


Objective:  To  plan,  coordinate,  conduct  and 
disseminate  the  results  of  STARS  human 
engineering  symposia  and  workshops. 


Status:  Agenda  for  strategy/coordination 

workshop  prepared  and  submitted  for  review 
and  comment. 


HUMAN  ENGINEERING  FOR  STARS 

1985  EFFORT 

TASK  4 

Scaling  of  Users,  Systems  and  Environments 


Objective:  To  review  the  state  of  knowledge  and 
develop  a  plan  for  scaling  of  system  users, 
systems  and  environments. 


Status:  Preliminary  definition  of  scaling 

objectives  and  applications  in  preparation. 
Scheduled  for  delivery  by  31  October  1985. 

Literature  search  initiated  on  scaling  methods 
and  applications. 
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TASK  4 

Scaling  of  Users,  Systems  and  Environments 

ACTIVITIES  AND  PRODUCTS  # 

ACTIVITIES:  1.  Identify  scaling  objectives  and 
applications  for  STARS  (in  progress) 

2.  Review  the  state-of-knowledge  for  user, 
system  and  environment  scaling  in 
computer-based  systems 

3.  Develop  methods  and  identify  criteria  for 
user,  system  and  environment  scaling  for 
STARS 

4.  Integrate  scaling  methods  and  criteria  into  0 
STARS-SEE  workstations  for  program 
managers,  systems  analysts  and  human 
engineers. 

PRODUCTS:  1.  A  description  of  the 

state-of-knowledge  for  user,  system  and 
environment  scaling  in  computer-based  systems 

2.  A  detailed  specification  describing  the 
objectives  and  application  of  scaling  methods 
and  criteria  for  complexity,  workload  and 
information  flow  in  computer-based  systems. 
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•  HUMAN  ENGINEERING  FOR  STARS 

1985  EFFORT 

TASK  5 

User-Workstation  Interface  Design 

Requirements 


Objective:  To  review  the  state  of  knowledge  and 
develop  an  R&D  plan  for  determining  STARS 
#  user-workstation  interface  requirements 


Status:  Preliminary  list  of  user  functions  and 
activities  prepared  and  submitted  for  review 
and  comment. 

Literature  search  initiated  on  workstation 
design  for  APSE  (Ada  Programming  Support 
Environment). 

Draft  SEE  user  questionnaire  prepared  and 
submitted  for  review  and  comment. 
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TASK  5 

User-Workstation  Interface  Design 

Requirements 

ACTIVITIES  AND  PRODUCTS 

ACTIVITIES:  1.  Review  the  state-of-knowledge 
for  user-workstation  interface  design 

2.  Define  range  of  functions  and  activities  of 
each  class  of  STARS-SEE  users,  focusing  on 
project  managers,  system  analysts, 
programmers,  and  human  engineers 

3.  Identify  the  number  of  unique  functions  and 
activities  of  STARS-SEE  users  and  describe 
their  associated  requirements  and  interfaces 

4.  Prepare  a  detailed  specification  for 
user-workstation  interface  requirements 
organized  by  user  class,  function  and  activity. 

PRODUCTS:  1.  A  description  of  the 

state-of-knowledge  for  user-workstation 
interface  design  requirements 

2.  A  detailed  specification  describing  the 
objectives  and  application  of  user-workstation 
design  requirements  for  each  user  class, 
function  and  activity. 


ATTACHMENT  M.3 


USER,  SYSTEM  AND  ENVIRONMENT  SCALING 


USER,  SYSTEM  &  ENVIRONMENT 

SCALING 


OBJECTIVE:  To  develop  and  validate  methods, 
measures  and  criteria  for  scaling  user 
characteristics,  system  usability  and 
environmental  complexity. 


USER,  SYSTEM  &  ENVIRONMENT  SCAUNG 
TASK  FLOW 


CANDIDATE  SCAUNG  PARAMETERS 

USERS 

•  Experience 

•  Skill/Knowledge  Levels 

•  Aptitudes 

SYSTEMS 

•  Operational  Complexity 

•  Usability 

•  Workload 

•  Error  Tolerance 

ENVIRONMENTS 

•  Operational  Complexity 

•  Workload 

•  Error  Tolerance 
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CANDIDATE  SCALING  MEASURES 

USERS 

EXPERIENCE 

•  Formal  Training 

•  OJT 

•  Systems  (number,  variety) 

SKILL/KNOWLEDGE  LEVELS 

•  Proficiency  Scores 

•  Supervisor  ratings 

•  Special  Tests 

APTITUDES 

•  Traditional  (e.g.,  mechanical,  clerical,  etc) 

•  Innovative  (e.g.,  multi-dimensional  spatial) 

CANDIDATE  SCALING  MEASURES 
SYSTEMS 

OPERATIONAL  COMPLEXITY 

•  Number  of  decisions 

•  Complexity  of  decisions 

•  Criticality  of  decisions 

USABILITY 

•  Availability  of  information 

•  Quality  of  information 

WORKLOAD 

•  Rate  of  decision  making 

ERROR  TOLERANCE 

•  Precision/accuracy  requirements 

•  Error  detection/recovery  procedures 
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CANDIDATE  SCAUNG  MEASURES 
ENVIRONMENTS 


WORKLOAD 

•  Rate  of  decision  making 

•  Criticality  of  decisions 

ERROR  TOLERANCE 

•  Threat  environment 

•  Precision/accuracy  requirements 

•  Error  detection/recovery  procedures 


KIRKPATRICK’S  CONTINUUM  OF  TASK  DIFFICULTY 


POURING  NOXIOUS  LIQUID 
OUT  OF  A  BOOT  WITH  THE 
INSTRUCTIONS  WRITTEN 
ON  THE HEEL 


ALL  OTHER  TASKS 


LANDING  THE  SPACE 
SHUTTLE  ON  A  CARRIER 
DECK  DURING  A  SEA 
STATE  NINE 


LOW 


TASK  DIFFICULTY 


HIGH 
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RELATIONSHIP  OF  SYSTEM/ENVIRONMENT  COMPLEXITY 
TO  USER  PERFORMANCE 


CANDIDATE  SCALING  TECHNIQUES 

SYSTEM  COMPLEXITY-TASK  DIFFICULTY 
DIRECT  MEASUREMENT-State  Transition  Diagrams 

•  NUMBER  OF  DECISION  POINTS  (Nodes) 

-  Per  Task 

-  Per  Screen 

-  Per  Unit  Time 

•  COMPLEXITY  OF  DECISION  POINTS  ( Nodes) 

-  Availability  of  information 

-  Quality  of  information 

•  NUMBER  OF  INPUT  REQUIREMENTS  (Arcs) 

-  Per  Task 

-  Per  Unit  Time 

•  CONSISTENCY  OF  INPUT  REQUIREMENTS  (Arcs) 

-  Within  tasks/modules 

-  Across  tasks/modules 


EXAMPLE  STATE  TRANSITION  DIAGRAM 


PROGRAM  MANAGER’S  ’TOOL  SCREEN" 


^  PROJECT  PLANMNG> 

7 "-f 

T  h 


PROJECT  CONTROL  > 
<$/V  PROTO TYFr*3> 


H  <S  /V  INTEGR  AT  ION /TEST  > 

f3  <CCt#")CU9 ATION  management > 

•4* 

<QUAlfTY  ASSURANCE > 


v,  -mM 


Actions  (6) 

1  call  shutdown 

2  call  startup 

Diagram  main  entry=select,  exlt="n",  "no** 

Message  headers  TOOLS  MENU" 

Message  lastline  »  "Press  "Q"  to  quit,  or 

"M"  to  return  to  main  menu" 

Node  SELECT 
-Please  Choose:" 

T.  Project  Planning" 

"2.  Project  Control" 

"3.  Software  Prototyping" 

"4.  Software  Integration  and  Testing" 

"5.  Configuration  Management" 

"6.  Quality  Assurance" 

"H(elp):  Information  on  use  of  program" 

"Q(ult):  End  session" 

"Your  choice: _ " 

Node  HELP 

"For  more  Information  about  a  command, 
enter  the  command  number,  " 

"followed  by  a  ?  and  press  'return'  " 

"or  type  "help"  or  "?"  while  In  the  command  mode." 

Node ERROR 
bell 

"Please  enter  the  approriate  number  (1-6)  to  select  a 
command," 

"Or  enter  "help"  or  "?"  for  help,  or  "quit"  to  end  session" 
Node  END 

"If  you  are  finished  and  would  (Ike  to  end  this  session,  enter 
"Yes"" 

To  return  to  the  main  menu,  enter  "No"" 

"End  session? _ " 

NodeX 
clear  screen 
return  to  OS 
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CANDIDATE  SCALING  TECHNIQUES 

SYSTEM  COMPLEXITY-TASK  DIFFICULTY 


INDIRECT  MEASUREMENT  -  Magnitude  Estimation 
APPROACH: 

1.  Develop  Task  "simulators"  with  adjustable 
level  of  difficulty  for  generic  tasks  (pattern 
recognition,  discrimination,  fine  motor,  etc.) 

2.  Develop  continuum  of  empirical  error 
probabilities  for  varying  levels  of  difficulty 
for  each  task 

3.  Have  experienced  operators  adjust 
simulators  to  equal  perceived  difficulty  for 
real-world  tasks  for  different  systems  and 
operating  environments 

4.  Extrapolate  error  probabilities/task 
difficulty  for  real-world  tasks,  systems  ans 
operating  environments 

5.  Develop  compendium  of  error  probabilities 
for  each  task  type,  system  and  operating 
environment 


ATTACHMENT  M.4 


USER-WORKSTATION  INTERFACE  REQUIREMENETS 


HUMAN  ENGINEERING  FOR  STARS 


MISSION:  To  develop  and  validate  human 
engineering  principles,  methods  and  criteria  that 
will  enhance  productivity  throughout  the  life 
cycle  of  mission-critical  computer  resources. 


KEY  PLAYERS: 

UCI  WORKING  GROUP 
STARS  HUMAN  ENGINEERING 
STANDARDIZATION  GROUP 
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HUMAN  ENGINEERING  FOR  STARS 

OBJECTIVES: 


1.  TO  MAXIMIZE  PRODUCTIVITY  OF  EMERGING 
ADA  PROGRAMMING  SUPPORT  ENVIRONMENTS 
(APSEs)  BY  FOCUSING  ON  THE  REQUIREMENTS  OF 
THE  SOFTWARE  PROFESSIONAL 


2.  TO  EXPAND  THE  EXPERIENCE  BASE  THROUGH 
THE  APPLICATION  OF  HUMAN  ENGINEERING 
TECHNOLOGY  TO  ACTUAL  DEVELOPMENT 
PROJECTS. 


3.  TO  EXPAND  THE  TECHNOLOGY  BASE  BY 
SUPPORTING  CONTINUED  DEVELOPMENT  OF 
HUMAN  ENGINEERING  KNOWLEDGE,  METHODS 
AND  TOOLS. 
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HUMAN  ENGINEERING  FOR  STARS 

STRATEGY: 


1.  SUPPORT  THE  SYSTEMATIC  APPLICATION  OF 
CURRENT  HUMAN  ENGINEERING  KNOWLEDGE  TO 
DESIGN  THE  USER  INTERFACE  FOR  THE 
AUTOMATED  PROGRAMMING  SUPPORT 
ENVIRONMENT. 

2.  SUPPORT  THE  SYSTEMATIC  APPLICATION  OF 
CURRENT  HUMAN  ENGINEERING  KNOWLEDGE  TO 
CONDUCT  EXPERIMENTS  TO  ESTABLISH  AND  TEST 
BASIC  PRINCIPLES  AND  MODELS  OF 
HUMAN-COMPUTER  INTERACTION  AND  PROBLEM 
SOLVING. 

3.  SUPPORT  THE  SYSTEMATIC  APPLICATION  OF 
CURRENT  HUMAN  ENGINEERING  KNOWLEDGE  TO 
PROVIDE  FOR  DEVELOPMENT  AND  CONTINUED 
ENHANCEMENT  OF  A  GENERAL  HUMAN 
ENGINEERING  METHODOLOGY  AND  ASSOCIATED 
SUPPORT  TOOLS. 


4.  SUPPORT  THE  SYSTEMATIC  APPLICATION  OF 
CURRENT  HUMAN  ENGINEERING  KNOWLEDGE  TO 
IDENTIFY  MEASURES  TO  ASSESS  PROGRESS. 
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HUMAN  ENGINEERING  FOR  STARS 


STARS  HUMAN  ENGINEERING  PROGRAM  PLAN 


OBJECTIVE:  To  review  the  state-of-knowledge  and  conduct 
research,  as  necessary,  to  achieve  the  objectives  of  the 
human  engineering  Program  Plan 

ACTIVITIES:  1. Outline  research  goals 

2.  Identify  and  select  support  contractors 

3.  Conduct  technology  surveys 

4.  Conduct  workshops  and  symposia 


5.  Conduct  special  studies 
PRODUCTS:  1.  Near-  and  long-term  research  goals 

2.  Reports  on  the  state-of-knowledge  for 
selected  topic  areas 

3.  Results  of  workshops,  symposia,  technology 
surveys  and  special  studies 
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[ - PHASE  1  -  PLANNING - ] 

OBJECTIVE:  To  develop  an  overall  STARS  human 
engineering  program  plan  that  integrates  the  various 
STARS  ACT  activities. 

ACTIVITIES:  1.  Prepare  project  plan 

2.  Review  STARS  Program  documentation 

3.  Prepare  human  engineering  Program  Plan 

PRODUCT:  A  detailed  Program  Plan  specifying  critical 
human  engineering  objectives,  activites  and  products 
associated  with  each  major  STARS  milestone. 


Draft  Program  Plan  focuses  on  six  areas: 

1.  Coordination  and  integration  of  STARS 
activities 

2.  User-computer  interface  requirements 
for  SEE  workstations 

3.  Design  of  training  systems  and  devices 
for  SEE 

4.  Human  factors  in  software  system 
acquisition 

5.  Methods  and  measures  for  user,  system 
and  environment  scaling 

6.  Methods  and  procedures  for  a  standard 
human  engineering  process  for  software 
development. 
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HUMAN  ENGINEERING  FOR  STARS 


[ -  PHASE  2  -  COORDINATION/INTEGRATION  - J 

OBJECTIVE:  To  assist  STARS  Area  Coordinating  Teams 
(ACTs)  in  coordinating  and  integrating  their 
technicai  activities  to  support  implementation  of 
the  human  engineering  Program  Plan. 

ACTIVITIES: 

1.  Conduct  coordination/strategy  meetings 
with  ACT  representatives 

2.  Review  ACT  objectives  and  plans 

3.  Identify  and  resolve  voids  and  duplications 

4.  Provide  ongoing  coordination/integration 

PRODUCT:  Implementation  of  the  human  engineering 
Program  Plan  that  ensures  reiated  activites  and 
products  are  coordinated  and  integrated  across  the 
five  ACTs. 


Coordination  and  integration  wlii  be  achieved 
through: 

1.  Workshops  and  symposia 

2.  Electronic  bulletin  boards,  newsletters, 
data  bases,  and  E-mail  networks 

3.  Report  dissemination 

4.  Periodic  status/coordination  meetings 
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HUMAN  ENGINEERING  FOR  STARS 


[ -  PHASE  3  -  RESEARCH  - ] 

OBJECTIVE:  To  review  the  state-of-knowledge  and  conduct 
research,  as  necessary,  to  achieve  the  objectives  of  the 
human  engineering  Program  Plan 


ACTIVITIES:  1. Outline  research  goals 

2.  Identify  and  select  support  contractors 

3.  Conduct  technology  surveys 

4.  Conduct  workshops  and  symposia 

5.  Conduct  special  studies 
PRODUCTS:  1.  Near-  and  long-term  research  goals 

2.  Reports  on  the  state-of-knowledge  for 
selected  topic  areas 

3.  Results  of  workshops,  symposia,  technology 
surveys  and  special  studies 
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I—  PHASE  4  -  REQUIREMENTS  SPECIFICATION  — ] 

OBJECTIVE:  To  prepare  detailed  human  engineering 
design  specifications  for  products  originating  from 
each  of  the  five  Area  Coordinating  Teams 


ACTIVITIES: 

1.  Develop  human  engineering  requirements  and 
prepare  specifications  for  each  of  the  five  area 
coordinating  teams  based  on  the  results  of  Phase  3 

2.  Provide  ongoing  program  support  to  interpret  and 
implement  requirements  and  specifications. 


PRODUCTS:  Human  engineering  requirements  and 
specifications  for  ACT  products. 


Human  engineering  requirements  and  specifications 
will  be  prepared  for: 

1.  User-workstation  interfaces  for  STARS-SEE 
workstations 

2.  Human  engineering  methods  and 
criteria  applicable  to  all  phases  of 
the  software  acquisition  process 

3.  Training  systems  and  devices  for 
SEE  users  and  product  end-users 

4.  Methods  and  criteria  for  scaling 
users,  systems  and  environments 

5.  UCI  rapid  prototyping  design  and 
evaluation  methods 
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ATTACHMENT  M.5 

UCI/STARS  WORKSTATION  INTEROPERABILITY  CONSORTIUM 


Relationship  to  UCI  subTAG 


Voluntary  and  cooperative  group  of  key 
industry  and  technical  society  professionals  in 
UCI  domain  supporting  goals  and  objectives  of 


UCI  SubTAG 
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SCOPE 


1.  Provide  forum  for  Information  exchange  and  liaison 
between  government,  technical  societies  and 
industry  groups  (e.g.,  TS/1C). 


2.  Explore  common  areas  of  current  need 


3.  Define  common  needs  and  objectives  -  UCI  tools, 
methods  and  references 


4.  Support  UCI  TAG  -  integrated,  cooperative  R&D 
efforts  In  areas  of  defined  needs 

5.  Develop  recommendation  for  consideration  by  UCI 
TAG  in  areas  of  UCI  -  STDs,  guidelines,  tools, 
methods,  ongoing  research 


GOALS 


1.  Reduce  training  needs  by  maximizing  the 
transferability  of  learning  among  interfaces 


2.  Exhance  operability  and  interoperability 


3.  Reduce  overall  system  life  cycle  costs 


4.  Increase  productivity 


5.  improve  system  readiness/effectiveness 


6.  Identify  research  voids  and  minimize  duplication  of 
effort 


7.  Extend  the  UCI  technology  base 
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TASKS 


1.  Support  workshops/symposia 

2.  Identify  existing  techinal  Information  sources 

3.  Conduct  studies  to  define  mutual  needs  for  UCI 

4.  Support  cooperative  R  &  D  efforts 

5.  Plan  and  conduct  R  &  D  efforts 

6.  Synthesize  findings  from  R&D  efforts  and 
provide  recommendations  for  additional 
research/future  advancements 


PRODUCTS 


Inputs  and  recommendations  for  symposia  and 
workshops 

Database  of  Industry  and  government  UCI 
resources 

Inputs  to  UCI  technology  needs  summary 

Research  reports  and  project  cards 

Inputs  to  near-term  and  long-term  research 
plans 
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ATTACHMENT  M.6 


COPE/RIPL 


COPE  TECHNOLOGY 


I 

I 


INTEGRATED  ANALYSIS  TECHNIQUES  FOR  C>  SYSTEMS 

ADAPTIVE  OPERATOR  AIDING 

INTEGRATED  O  WORKSTATION  CONCEPTS 

TOUCH-SENSITIVE  DISPLAY  INVESTIGATIONS 

DECISION-MAKING  IN  O 

GROUP  PROBLEM  SOLVING 

LARGE  DISPLAY  EVALUATIONS 

HCI  DESIGNER  SUPPORT  STATION 


M.6-1 


INTEGRATED  ANALYSIS  TECHNIQUES 

IAT 


AIR  FORCE  AEROSPACE  MEDICAL  RESEARCH  LABORATORY 
INTEGRATED  ANALYSIS  TECHNIQUES 


PROBLEM 

—  HOW  DO  YOU  DESCRIBE  THE  HUMAN  ELEMENT  IN  O  SYSTEMS 
—  WHAT  ARE  HUMAN  MOP  AND  HOW  DO  THEY  RELATE  TO  SYSTEM  MOE 

OBJECTIVE 

—  INCLUDE  HUMAN  PERFORMANCE  IN  A  STRUCTURED  ANALYSIS  AND  DESIGN  PROCESS 
—  DEVELOP  FRAMEWORK  FOR  MAN-IN-THE-LOOP  SIMULATION  AND  DATA  COLLECTION 

APPROACH 

—  AUGMENT  IDEF0  TECHNIQUE  TO  FORM  A  SYSTEM  DESCRIPTION  FRAMEWORK 

-  INTERFACE  TO  SAINT 

-  CONDUCT  EXPERIMENTATION  TO  DEVELOP  HUMAN  PERFORMANCE  DATA  BASE 
STATUS. 

—  IDEF0  applied  at  norad  W'TH  EXCELLENt  SUCCESS 
—  INTERFACE  WITH  SAINT  HAS  .BEEN  TESTED  ON  A  THREAT  SYSTEM  SIMULATOR 
—  OPERATIONAL  MEASURES  BEING  DEVELOPED  FROM  NOUAD  STUDIES 
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INTEGRATED  ANALYSIS  TECHNIQUES 


DIMENSIONS  FOR  ANALYZING  O  SYSTEM  STRUCTURE 


1.  GOALS  -  CAPTURE  THE  REQUIREMENTS/CONSTRAINTS  THAT 

MISSION  OBJECTIVES  IMPOSE 

2.  ORGANIZATIONS  -  SITE  OF  CONTROL  OR  AUTHORITY:  PERSONNEL  PLAY 

ROLES  AT  DIFFERENT  LEVELS  OF  ORGANIZATION.  AND 
HAVE  RESPONSIBILITIES  ALLOCATED  TO  THEM 

3.  PROCESSES  -  ACTIVITIES  CARRIED  OUT  TO  FULFILL  GOALS 

H.  RESOURCES  -  AGENTS  (MAN  OR  MACHINE):  EQUIPMENT.  MATERIALS. 

PHYSICAL  LAYOUT  OF  COMPONENTS  AND  THEIR  CONNEC¬ 
TIONS 


ISSUES  RELEVANT  TO  ADAPTIVE  AIDING 


FOCUS  OF  ADAPTATION 


METHOD  OF  ADAPTATION 


NATURE  OF  COMMUNICATION 


WHO'S  IN  CHARGE? 


COMPUTER  DECISION  MAKING 


GOALS  OF  RESEARCH 


•  IDENTIFY  PRINCIPLES  OF  ADAPTIVE  AIDING 


•  DETERMINE  CONDITIONS  IN  WHICH  HUMANS  CAN 
BE  IN  CHARGE 


•  PROVIDE  EXPERIMENTAL  RESULTS  TO  SERVE  AS 
BASES  FOR  DESIGN  GUIDELINES 
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COPE  PROTOTYPE  WORKSTATION 


PERFORMANCE  ISSUES: 

•  DISPLAY  FORMATS 

•  DATA  ENTRY  (VOICE,  TOUCH) 

•  VOICE  SYNTHESIS  (PROMPTS) 

•  COMMUNICATION  MANAGEMENT 

•  STATION  FLEXIBILITY 

•  PHYSICAL  DESIGN  (PROFILE,  LIGHTING) 


COPE  TECHNOLOGY  TRANSITION 


t  SPACECMD/NORAD  EVALUATIONS  AND  DESIGN  RECOMMENDATIONS 
t  SOPC-- IDENTIFIED  PROBLEM  AREAS  FOR  PROPOSED  SOPC  DESIGN 
t  JOINT  TACTICAL  FUSION  PROGRAM-HUMAN  FACTORS  RECOMMENDATIONS 

•  NATIONAL  MILITARY  INTELLIGENCE  CENTER  EVALUATION  AND  RECOMMENDATIONS 
t  OJCS— COMMAND  CENTER  IMPROVEMENT  PROGRAM 

t  TOUCH  SENSITIVE  DISPLAY  GUIDELINES  FOR  O  APPLICATIONS 
t  RECOMMENDATIONS  FOR  APPLICATION  OF  LARGE  DISPLAYS 

•  SPACECMD  ADVANCED  INTEGRATED  WORKSTATION  DEMONSTRATION 
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OBJECTIVES: 

•  DEVELOP  TECHNIQUES  TO  DESCRIBE  AND 
ANALYZE  C3  SYSTEMS 

•  CHARACTERIZE  HUMAN  PERFORMANCE  IN  C3 
SYSTEMS 

•  ESTABLISH  GUIDELINES  FOR  C3  STATION 
DESIGN 

MILESTONES: 

•  ADVANCED  INTEGRATED  C3  WORKSTATION  1985 

•  USERS  MANUAL  FOR  C3  SYSTEM  ANALYSIS 

METHODOLOGY  1986 

•  APPLICATIONS  OF  LARGE  DISPLAYS  TO  C3 

SYSTEMS  1987 

•  NIMOHt!  ADAMtIVI  AIDING  POM  Ca  IBS? 

•  INTEGRATED  ANALYSIS  TECHNIQUES  FOR  C3 

SYSTEMS  1988 

•  HANDBOOK  OF  C3  WORKSTATION  DESIGN  1988 
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HCI  DESIGNER'S  SUPPORT  STATION 
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HCI  DESIGNER  TEMPLATES 


RIIPL 

RAPID  INTELLIGENT  PROTOTYPING  LABORATORY 


DEFinmon 


RIIPL 


R  apid 
1  ntelligent 
P  rototyping 
L  aboratory 


-  EASY  TO  LLAIIN  AND  USE 

-  CONTAINS  USI  GUIDELINE  CXPERI ISE 

-  SIMULATES  Till:  USI  OF  AN  APPLICATION 

-  A  HARDWARE  AND  SOI  TWAIN;  ENVIRONMENT 


TECHfllCHL  APPROACH 


RIIPL 


I)  EXPERT  SYSTEM  -  RIPL  Incorporates  two  expert  systems: 

•  a  Mycln  like  consultant  expert  system  that  responds  to  general  USI  Information 
requests  (e.g.  Tell  me  about  tabular  data  presentation)  ;  and 

•  an  evaluation  expert  system  that  Interprets  the  current  definition  of  dialogues 
and  displays  (e.g.  Use  of  yellow  on  top  of  orange  Is  not  recommended). 


2)  KNOWLEDGE  BASE  -  RIPL  Incorporates  the  Smith  6*  hosier  guidelines  In  three 
ways: 

•  In  a  “help  by  example*  facility; 

•  In  the  Implementation  of  It  I  I’M  l.c,  limit  the  user's  options);  and 

•  In  the  USI  expert  systems  knowledge  base  (same  rule  base  for  both  experts). 


3)  PROTOTYPE  SEMANTICS  -  RIPL  combines  a  direct  manipulation  user  Interface 
with  a  graphical  ellipsis  dlaloquc  Interpreter  to  minimize  the  prototype  and  semantic 
definition  overhead. 


4)  HARDWARE  SUITE  -  Use  of  the  VAX  architecture  and  the  CK5  standard  allows  a 
wide  range  of  upward  migration  paths  for  processing  power  and  display  capability. 
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PROTOTYPIFIG 


R II  I?  L 


MAJOR  FUNCTIONS 


RIIPL 


EXECUTIVE 

S«ts  up  all  necessary  file  and  library  accesses. 

Performs  housekeeping  functions  of  deleting,  renaming  and  backing  up 
RIPL  files  and  libraries. 

PROTOTYPE  BUILD  SUBSYSTEM 

Defines  screen  contents,  interactions,  and  target  environment. 
Instruments  dialogue. 

SIMULATION  SUBSYSTEM 

Compiles  simulation  components. 

Simulates,  records,  and  measures  the  USI  design. 

USI  ADVISORY  SUBSYSTEM 

Consults  on  and/or  evaluates  design. 

TECHNICAL  LIBRARIAN 

Provides  for  browsing  of  guidelines  and  examples. 

Provides  help  on  using  RIPL. 
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INPUTS  &  OUTPUTS 


RIIIPL 


INPUT: 


1  m<?Pl  AY  rT>NTFNT 

2.  PHYSICAL  OPERATINO  ENVIRONMENT 

3.  DIALOGUE  STRUCTURE 

4.  SIMULATION  PARAMETERS 

5.  APPLICATION  OPERATINO  ENVIRONMENT 


OUTPUT: 


1.  STRUCTURED  DIALOGUE  DEFINITION 

2.  DESION  CONSULTATION 

3.  DESION  EVALUATION 

4.  DESION  SIMULATION 

5.  CAPTURE  OF  USER  INTERACTION  DURING  SIMULATION 

6.  INSTRUMENT  MEASURES 


TERMINOLOGY 


RIIIPL 


STATICS 

o  lik  -  a  rectangular  area,  less  than  or  equal  to  a  screen  in  a  size,  having 
specific  attributes  and  contents.  A  tile  is  similar  to  the  graphics  term 
viewport,  except  that  there  is  no  notion  of  a  larger  virtual  screen  area 
behind  it.  Tiles  may  be  grouped  in  a  hierarchy  where  tiles  that  are 
contained  in  the  area  defined  by  larger  tiles  inherit  some  of  their 
attributes.  The  attributes  of  a  tile  are  dependent  on  its  position  in  the 
hierarchy'. 

o  mosaic  -  the  contents  of  a  user  activity  screen  on  a  single  workstation 
at  any  one  time.  Each  mosaic  is  composed  of  a  hierarchy  of 
tiles. 

o  gallery  -  the  collection  of  all  the  user  activity  screens  on  all  displays  at 
anyone  time. 

DYNAMICS. 

o  thread  -  a  single  chain  of  stimulus-responses  originating  with  the 
simultaneous  activation  of  a  set  of  tiles. 
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TERM  HIERARCHY 


RIIPL 


USI  PROTOTYPE  SIMULATION  . . \ 


V. 


J 


TILE  DEFINITION 


riipl 


o  IDENTIFICATION  -  NAME  +  ID  NUMBER 


o  CONTENT  -  ( text/ table/ me nu/bit  map/display  list ...) 


o  SEMANTIC  USE  -  (error,  map,  title,  status  ...) 

o  LIST  OF  ATTRIBUTE  INSTANCES  (position,  size,  border,  view,  ounor  ...) 
o  LIST  OF  CONTENT  INSTANCES 
o  LIST  OF  STIMULUS-RESPONSES 
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DIALOOUE  THREAD 


RIIPL 


SUrt  Ll»t  Til!  t 


RIPL  TECHNICAL  LIBRARIAN 


riipl 


o  PROVIDES  ON-LINE  DOCUMENT  ACCESS 
o  FACILITATES  RAPID  INFORMATION  SEARCH 
o  FUNCTIONS  AS  RIPL  HELP  SYSTEM 
o  FUNCTIONS  AS  USI  GUIDELINE  RETRIEVAL  SYSTEM 
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HflRDWflRE  SUITE 


riipl 


Dot 

Matrix 

Printer 


800Kb 

Floppy 


VAX5TATION  I 

2  Mbyte  Memory 

MtcroYMS,  Window  Mgr. 
GKS.Pascol.l  Isp 


OECnet 

VAX  1  1/780 

Pascal,  Lisp 

IBM  PCX! 
Color 

TEK  4I05& 
YT  100 

1 


TEK  40Ix& 
YTIOO 


TEK  40lx& 
YTIOO 


Digitizer 

TobWt 


Touch 

Screen 


Approach 


1  Select  a  £lu^s_of  human-computer  interaction  problems. 

2  Develop  n  standard  task  for  this  problem  area 

5  Develop  several  alternative  strategies  for  solving  the 
task 

Develop  task  network  models  for  these  strategies. 

Set  model  parameters  via  several  techniques. 

().  Collect  laboratory  data  on  subjects  performing  the  tasks 
using  the  modeled  strategies. 

7  Compare  model  predictions  to  actual  data. 
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HCI  DESIGNER'S  SUPPORT  STATION 


± ± 
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HCI  DESIGNER  TEMPLATES 


ATTACHMENT  M.7 


IDB  DEMONSTRATION  INSTRUCTIONS 


INSTRUCTIONS  FOR  DIALING  UP  THE 
THE  IDB  DEMO 

YOU:  DIAL  (614)  424-5850  (300  Baud)*  ;  when  your  modem  completes  the  connect, 

or  proceed 

DIAL  (614)  424-5450  (1200  Baud)* 

YOU:  <CRXCRXCR>. . .  ;  HIT  <CR>  several  times  until  VAX  responds 

VAX:  XX-XX  IS4000 

VAX:  DESTINATION? 

YOU:  vms<CR>  ;  enter  "vms" 

VAX:  PAUSE 

VAX:  CONNECTED  TO  XX-XX 

YOU:  <CRXCRXCR>. . .  ;  Hit  <CR>  key  several  times  until 

you  get  a  USERNAME  prompt 

VAX:  USERNAME 

YOU:  demodb<CR>  ;  enter  this 

VAX:  PASSWORD 

YOU:  Interface<CR>  -.enter  this. 

VAX:  ACCOUNT  # 

YOU:  1014283  ;  enter  this.  After  a  pause  the  IDB  demo  begins 

VAX:  WELCOME  TO  VAX/VMS... 

IDB:  USERNAME 

YOU:  demodb<CR>  ;  enter  this 

IDB:  PASSWORD 

YOU:  Interface<CR>  ;  enter  this,  you  should  now  be  in  IDB 

NOTE:  The  only  legal  search  term  is  "ABBREVIATIONS. 

If  you  have  any  questions  please  call  Lou  Myers  (614)  424-7166 
*(8  Bits,  No  Parity,  1  Stop  Bit,  ECH0=N) 
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ATTACHMENT  N 


VOICE-INTERACTIVE  SYSTEMS  — 

LIST  AND  AGENDA 


ATTENDEE 


DEPARTMENT  OF  DEFENSE 


HUMAN  FACTORS  ENGINEERING 
TECHNICAL  ADVISORY  GROUP 
VO ICE- INTERACTIVE  SYSTEMS 
5  November  1985/San  Diego,  California 


Reg i stered  At 


Mr.  Howard  Arnoff 

HQS  Naval  Air  Systems  Command 

Code  AIR  5313 

Washington,  DC  20361-5310 

<202)  692-7488 

222-7438 


Ms.  Suzanne  V.  Bemis 
Naval  Ocean  Systems  Center 
271  Catalina  Blvd,  Code  441 
San  Dieoo,  CA  92152-5000 
<619)  225-7372 
933-7372 


Major  Randy  B  i  gum 
HQ  TAC/DRFA 
Lang  1 ey  AFB , 

VA  23665-5001 
<304)  764-5279 
432-5279 


endee  List 


Dr .  Thomas  M.  Cook 

Martin  Marietta  Aerospace.  M.S. 

P . 0 .  Box  1 79 

Denver  ,  CO  30210 

•303)  971-5698 

1M DA  3 US C  - 1  S I  Attn:  T  .  M  .  C o  o  k 


Mr.  Eugene  C.  Crist 
USNSWC/DL,  Code  N32 
Bldg.  1500,  Rm.  227 
Dahlgren,  VA  22448 
<703)  663-4651 
249-4651 


Dr  .  Tice  DeYoung 
SPAWARSYSCOM 
Code  6131 

Washington,  DC  20363-5100 
<202)  692-9207/9203 


Ms .  K .  Christ 
USA  HEL 

ATTN:  SLCHE-AD  <K.  Christ) 

Aberdeen  Proving  Ground,  MD  21005-5001 

<301)  278-5836/5837 

298-5836/5837 


Dr.  Jeffery  Ellman 
Phonetics  Laboratory 
University  of  California  at  San 
San  Diego,  CA  92152 


Mr .  Cl  ay  ton  Col er 
NASA-Ames  Research  Center 
MS  239-3 

Moffett  Field,  CA  94035 
<415)  694-5716 
FTS  464-5716 


Mr  .  Joe  M .  Jones 
AD/ENP 

Egl i n  AFB,  FL  32542 
<904)  882-8652 
872-8652 


N-l 


L8010 


D  i  ego 


Mr .  Mark  kan  ter 
NJ'cC 

3512,  Bldg.  1171-1 
Newport,  RI  02841-5047 
(401)  841-2643 
948-2643 


Mr.  Luis  E.  Posada 
NTSC 


Code  224 

Orlando,  FL  32813-7100 
(305)  646-4486/4349 
791-4436/4849 


Mr.  Warren  G.  Lewis 
Code  441 

Naval  Ocean  Systems  Center 
San  Diego,  CA  92152 
(619)  225-7372 


Mr .  Lockwood  Reed 

U.S.  Army  Avionics  R&D  Activity 

SAVAA-C 

Ft.  Monmouth,  NJ  07703 
(201)  544-2559 
995-2559 


Mr.  Gordon  K.  McElroy 
Code  3152 

Naval  Weapons  Center 
China  Lake,  CA  93555-6001 
(619)  939  -3167 
437-3167 

MCELR0Y3NWC-1 43B 


Mr.  Lloyd  Roberts 

Army  Foreign  Science  &  Tech.  Center, 
ATTN : AI AST-CB4 
220  E.  Seventh  Street 
Charlottesville,  UA  22901 
(804)  296-5171  x  433 
274-7433 


Lt.  Jack  D.  Mohney 
HQ  BMO/AWRH 
Norton  AFB,  CA  92409 
(714)  382-7067 
876-70o7 


Mr.  D.  Reed  Morgan 
AFWAL/AAAS— 2 

Wr i oh t -Pa t ter son  AFB,  OH  45433 

(513)  255-4827 

285-4827 


Mr.  Steve  Nunn 

Naval  Ocean  Systems  Center 

Code  441 

San  Diego,  CA  92037 
(619)  225-7372 


Mr.  Juan  R.  Pastrana 
USA  AVRADA 
ATTN:  SAOAA-C 

Ft.  Monmouth,  NJ  07703-5401 

(201)  544-4896 

995-4896 


Ms.  Elaine  Schiller 
Naval  Ocean  Systems 
Code  441 

San  Diego,  CA  92152 
(619)  225-7372 
933-7372 


Cen  ter 


LCDR  Timothy  J.  Singer,  MSC  USN 
Code  60B5 

Naval  Air  Development  Cetner 
Warminster,  PA  18974-5000 
(215)  441-2134 
441-2134 


Mr.  Steve  Smith 

Rome  Air  Development  Center 

RADC/IRAA 

Gri-f-fiss  AFB,  NY  13441-5700 

(315)  330-4024 

587-4024 


Dr  .  Norman  Warner 
NADC 

Code  6021 

Warminster,  PA  18974-5000 

(215)  441-2947 

441-2947 


Mr .  Dav i d 
AFWAL/F 1 GR 


Willi amson 


Wr i gh t-Pat ter  son ,  AFB  OH  45433 
(513)  255-6696 
785-6 696 
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AGENDA 


4 


VOICE  INTERACTIVE  SYSTEMS  SUBTAG  MEETING 
5  November  1985 

1330  -  1625  Teak  Room,  Kona  Kal  Resort,  San  Diego 

WELCOME  AND  ANNOUNCEMENTS 
Clay  Coler,  Chairman,  Voice  SubTAG 

BUSINESS  MEETING: 

ROADMAPS,  ARPANET,  AND  OTHER  BUSINESS 

Presentation:  NOSC’S  ROLE  IN  SPEECH  RECOGNITION  ANO  SYNTHESIS 
Elaine  Schiller,  Naval  Ocean  Systems  Center,  San  Diego 

-  Coffee  Break  - 

Presentation:  AN  ARCHITECTURE  FOR  PARALLEL  PROCESSING  IN  SPEECH  RECOGNITION:  THE  TRACE  MODEL 
Jeff  Ellman,  Phonetics  Laboratory,  University  of  California  at  San  Diego 

Presentation:  INTERFERENCE  REDUCTION 
Stephen  Smith,  Rome  Air  Development  Center,  Grlfflss  AFB,  Rome,  New  York 


AGENDA 

VOICE  INTERACTIVE  SYSTEMS  SUBTAG  MEETING 
8  November  1985 

0830  -  1200  Naval  Ocean  Systems  Center,  San  Diego 
WELCOME 

Clay  Coler,  Chairman,  Voice  SubTAG 

Presentation:  VOICE  INTERACTIVE  AVIONICS  (VIA)  FLIGHT  TEST  PREPARATIONS 
Lockwood  Reed,  U.S.  Army  Avionics  RID  Activity,  Ft.  Monmouth,  New  Jersey 

Presentation:  SOFTWARE  METHODOLOGY  FOR  AUTOMATED  RECOGNITION  TRAINING  (SMART) 

David  Williamson,  AFWAL/FIGR,  Wrlght-Patterson  AFB,  Ohio 

Presentation:  VOICE  RESEARCH  AT  THE  HUMAN  ENGINEERING  LABORATORY 
Kathy  Christ,  U.S.  Army  Human  Engineering  Laboratory,  Aberdeen  Proving  6round,  Maryland 

-  Coffee  Break  - 

Presentation:  APPLICATION  OF  CONNECTED  SPEECH  RECOGNITION  TO  TACTICAL  VOICE  COWAND  AND  SPEAKER  AUTHENTICATION 
Robert  Wohlford  and  8.  Patrick  Landell,  ITT  Defense  Communications  Division,  San  Diego 

Presentation:  DARPA  SPEECH  RESEARCH 
Tice  DeYoung,  SPAWARSYSCOM,  Washington,  D.C. 
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ATTACHMENT  0 


HUMAN  ENGINEERING  GUIDE  TO  SYSTEMS  AND  EQUIPMENT 


DEVELOPMENT 


VIEW  GRAPHS 


STATUS:  NRC/NAS  RECOMMENDATIONS 


•  NRC  CONSIDERS  NEED  TO  REVISE  HEGED  BY  REQUEST...  OCTOBER  1983 


•  NRC  CANNOT  UNDERTAKE  OR  SUPERVISE  REVISION 


RESERVATIONS  RE:  PRINTED  HANDBOOK  FORMAT... 


SUGGESTS  CONSIDERATION  OF  ALTERNATIVE  ELECTRONIC  FORMAT 


-  TELEPHONE  ACCESSIBLE 

-  CASE  OF  UPDATE 


o-i 


I NTEGRATED 
P  ERCEPTUAL 
I  NFORMATION 
FOR 

Designers 


lpi 

1 D  INFORMATION  MANAGEMENT  OBJECTIVES 


PRODUCTS 

•  CONSOLIDATION  - -  HANDBOOK 

•  PRESENTATION  - -  DATA  COMPENDIUM 

•  ACCESSIBILITY  - -  AUTOMATED  DATA  BASE 

MANAGEMENT  SYSTEM 
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I  PI D  STATUS 


•  HANDBOOK  OF  PERCEPTION  AND  HUMMPERFMMANCE 

-  PUBLICATION:  MAR-APR  1986 

-  95  CHAPTERS;  3000  PAGES;  2  VOL 

-  BOFF,  K.R.,  KAUFMAN,  L.,  AND  THOMAS  J.  (EDS.).  HANDBOOK  OF  PERCEPTION 

m  HUMAN  PERFORMANCE.  JOHN  WILEY  8  SONS,  NEW  YORK.  1985. 

•  ENGINEERING  DATA  COMPENDIUM:  HUMAN  PERCEPTION  AND  PERFORMANCE 

-  PROTOTYPE  DEVELOPED  AND  DISTRIBUTED:  APRIL  1985 

-  EXPECTED  COMPLETION:  SPRING  1986 

-  3  VOLUMES:  1200  ENTRIES 


DESIGNERS  ASSOCIATE 


•  CHARACTERIZE  OPTIMAL  DoD  DESIGN  PROCESS 

•  DEFINE  TOOLS  AND  DATA  BASES  NEEDED 

•  CONFIGURE  WORK  STATION  CONCEPT 
—  CONDUCT  TRADEOFF  ANALYSIS 

-  BLUEPRINT  PERSPECTIVE  ARCHITECTURES 
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DESIGNER’S  ASSOCIATE 


AUTOMATED  DATA  MANAGEMENT  FOR  DESIGNERS 

•  ENCODES 

—  I  PI  D  HUMAN  PERFORMANCE  DATA 
—  BIOTECHNOLOGY  AND  SYSTEMS  DESIGN  DATA 

•  FUNCTIONS  AS  A  “PROBLEM-SOLVING”  PARTNER 

-  USER  FRIENDLY 

-  HELPS  FORMULATE  RELEVANT  QUESTIONS 
—  TAILORS  QUESTIONS  TO  INFORMATION 

RESOURCES 

—  HELPS  DEVISE  RETRIEVAL  STRATEGIES 
—  EXPLAINS  AND  QUALIFIES  ANSWERS  THAT 
COME  BACK 
—  DECISION  AIDING 


HUMAN-PERCEPTION  AND  PERFORMANCE  SHORT  COURSE 


•  OBJECTIVE:  PROVIDE  SYSTEM  DESIGNERS  A  HUMAN  PERFORMANCE  FRAMEWORK 

FOR  DECOMPOSING  EQUIPMENT  RELATED  DESIGN  PROBLEMS  • 

•  WHEN:  APRIL  7-11,  1986 

«  WHERE:  BERGAMO  CONFERENCE  CENTER,  DAYTON  OHIO 


•  ENROLLMENT:  100  PARTICIPANTS  (50  GOVERNMENT,  50  INDUSTRY) 

•  FOR  FURTHER  INFORMATION 

CONTACT:  DR.  KENNETH  R.  BOFF 

(513)  255-7596  OR  7602 
AV  785-7596  or  7602 


0-4 


ATTACHMENT  P 


HUMAN  PERFORMANCE  MEASUREMENTS 


TUTORIAL 


OVERVIEW 


Department  of  Defense 


p-i 


HFE  Testing  and  Research 


PERFORMANCE 
MEASURES  TUTORIAL 

OVERVIEW 

•  Introduction 

—  Why  Is  the  Subject  Important? 

—  What  Are  the  Theoretical  Issues? 

•  The  Army  Experience 

•  The  Navy  Experience 

•  The  Air  Force  Experience 

•  The  Industry  Experience 

•  Open  Discussion 

•  Summary  and  References 


PANELISTS 

Army  —  James  C.  Geddie,  PhD 
Navy  —  Dennis  K.  McBride,  PhD 
Air  Force  —  Stephen  M.  Rokicki,  PhD 
APA/21  —  Frederick  A.  Muckier,  PhD 

MODERATOR 

John  L  Miles,  Jr.,  J.D. 


PERFORMANCE 
MEASURES  TUTORIAL 

WHAT  IS  A  "PERFORMANCE  MEASURE"? 

NO  ANSWER  IN: 

•  A  Comprehensive  Dictionary  of  Psychological  and 
Psychoanalytical  Terms 

•  Dictionary  of  Psychology 

•  Dictionary  of  Weapons  and  Military  Terms 

•  Handbook  of  Evaluation  Research 

•  Handbook  of  Psychological  Terms 

•  Modern  Science  Dictionary 


WHAT'S  THE  PURPOSE 
OF  A  "PERFORMANCE  MEASURE"? 

To  Select  from  the  Many  Facets 
of  Observable  Activity  One 
Which  Is  Assessable  in  Relevant, 
Meaningful  Quantifiable  Units. 


WHY  DOES  IT  MATTER  HOW  WE  SELECT 
"PERFORMANCE  MEASURES"? 

•  Validity  of  Our  Technology 

•  Communication  of  Our  Findings 

•  Construction  of  Data  Bases 


PERFORMANCE 
MEASURES  TUTORIAL 


WHAT  IS  THE  ONE,  BEST,  UNIVERSAL 
"PERFORMANCE  MEASURE"? 


(We  Haven't  Found  It  Yet.) 


MOST  COMMONLY  USED 
HUMAN  PERFORMANCE  MEASURES 

—  Time 

—  Accuracy* 


•For  Convenience,  We  Customarily  Measure  Error,  Assuming  Accuracy* 


Error 


MANPRINT  RULE  THREE 

Measure  Soldier  Performance 
by  Time  and  Accuracy 


PERFORMANCE 
MEASURES  TUTORIAL 

MANPRINT  CONCEPT  OF  "SYSTEM"  EFFECTIVENESS 


APTITUDES 

REQUIRED 


i 


1  HAVE 

INITIALIZED  IT 
PROPERLY 


%  CORRECT 
TIME  TO 
INITIALIZE 


OPERATOR 


COGNITIVE 

PROCESSES 


2  "1  AM  LOCATED 

HERE  ..." 


%  CORRECT 

3  IT  S  WORKING  TIME  TO  DETECT 
CORRECTLY  CHANGE  GIVEN 
CHANGE  OCCURS 


ACCURACY 

TIME 


TRAINING 
REQUIRED  FOR 
THOSE  APTITUDE 
CONFIGURATIONS 


I  CAN  FIX  IT.  ' 


%  CORRECT 
REPAIR  TIME 


WHAT  IS  THE  IDEAL  SOURCE 
OF  "PERFORMANCE  MEASUREMENT"? 


The  System  Requirements  Document  Contains: 


\  •  Soldier  Performance  Standards 


Soldier  Characteristics 


•  Maximum  Tolerable  Training  Burden 


COMMON  PROBLEMS  WITH 
"PERFORMANCE  MEASURES” 

•  Set  Too  Broadly 

•  Set  Too  Narrowly 

•  Fails  to  Discriminate 

•  Unrelated  to  Military  Concerns 


ATTACHMENT  Q 


BIBLIOGRAPHY  ON  PERFORMANCE  MEASUREMENT 


Department  of  Defense 
Washington,  DC  20310 


BIBLIOGRAPHY 

ON 

PERFORMANCE  MEASUREMENT 


HUMAN  FACTORS  ENGINEERING 
Technical  Advisory  Group 


November,  1985 


Part  One 


Analyzing  Training  Effectiveness  (1976).  (Report  No.  TRADOC  PAM 

71-8).  Headquarters  U.S.  Array,  Training  and  Doctrine  Command. 

Askren,  V.B.,  &  Regullnskl,  T.L.  (1983).  Quantifying  human 

performance  for  reliability  analysis  of  systems  (Report  No.  AD 
A134789).  Alexandria,  VA:  Defense  Technical  Information  Center, 
Defense  Logistics  Agency. 

Askren,  W.B.,  &  Regullnskl,  T.L.  (1969).  Mathematical  modeling  of 

human  performance  errors  for  reliability  analysis  of  systems  (Report 
No.  AD  687084).  Alexandria,  VA:  Defense  Technical  Information 
Center,  Defense  Logistics  Agency. 
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U_(4),  393-396. 
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analyzing  human  performance  data  In  a  materiel  development  project 
(Technlcaf  Memorandum  29-76).  Aberdeen  Proving  Ground,  MD:  U.S. 
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•  DYNAMIC  MAN-SYSTEM  AND  GROUP  INTERACTION 
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SOME  PRACTICE 


•  22  OPERATIONAL  SYSTEM  TESTS  IN  LAST  THREE  YEARS 

•  IN  EVERY  ONE,  PART  OF  PROBLEM  WAS  "MEASUREMENT” 
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(4.)  Evaluation,  not  diagnosis,  stressed 
(5.)  Measurement  "Acceptability” 

(6.)  Strong  demand  for  "Expert  Judgement" 

(7.)  Negativity  toward  Human  Performance  Measurement 
(8.)  Training  of  Testers  and  Evaluators 
(9.)  Instrumentation  always  different 
(10.)  Automated,  Computer-Aided,  System  PM 

•  WHAT  DO  WE  DO  IN  PRACTICE?  MUDDLE  THROUGH 


SOME  THEORY 


•  HOW  CAN  WE  IMPROVE  PERFORMANCE  MEASUREMENT? 

•  IF  WE  HAVE  QUANTITATIVE  THEORY _ 

—  THE  MEASURE  SETS  ARE  GIVEN 

•  IF  WE  DON'T.  NEED  A  METHOD  .... 

—  GOING  TO  TRY  ONE  ON  YOU 

•  A  10-STEP  MODEL  FOR  PERFORMANCE  MEASUREMENT 
DEVELOPMENT 

—  PROBABLY  NO  ONE  WILL  USE  IT 
—  BUT.  IF  PEOPLE  WILL  START  THINKING 

•  THE  BASIC  QUESTION: 

—  WHAT  DO  YOU  WANT  TO  KNOW? 
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A  MODEL  FOR  PERFORMANCE  MEASUREMENT  DEVELOPMENT 


WHAT? 


HOW? 


A  MODEL  FOR  PERFORMANCE  MEASUREMENT  DEVELOPMENT 

(continued) 
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ATTACHMENT  V 


IMPLEMENTATION  EFFORTS  FOR  MANPRINT 


MANPRINT 
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JUDAH  KATZNELSON 
U  S.  ARMY  RESEARCH  INSTITUTE 
FOR  THE  BEHAVIORAL  AND  SOCIAL 
SCIENCES 


MANPRINT  IN  THE  REAL  WORLD:  SIX  WAYS  FOR  A 
LOGISTICIAN  TO  EMBARRASS  A  HMPT  SPECIALIST 


For  the  past  several  years,  I  have  been  collecting  a  list.  The 
list  represents  all  the  frequently  well-intentioned,  but  usually 
critical,  comments  that  we,  in  the  people  business,  encounter  when  we 
try  to  introduce  Manpower  and  Personnel  Integration  (MANPRINT)  elements 
i.e.,  human  engineering,  manpower,  personnel,  training  (HMPT)  into  new 
design  environments.  So  far,  Ifve  gotten  up  to  six  -  feel  free  to  add 
to  my  list.  My  purposes  for  writing  this  paper  are  a)  to  communicate 
why  I  believe  we  have  matured  as  a  discipline  to  the  point  where  we 
don't  need  to  feel  embarrassed  or  take  a  defensive  posture;  and  b)  share 
some  "lessons  learned"  from  the  field  where  I  interacted  with  people 
representing  some  of  the  key  disciplines  in  the  weapons  systems 
acquisition  process.  The  Light  Helicopter-Experimental  (LHX)  was  the 
first  time  that  a  major  Army  weapon  system  acquisition  was  to  be 
procured  based  upon  a  performance-driven  Request  for  Proposal  (RFP).  No 
longer  was  the  government  going  to  dictate  a  detailed  "how  to".  The 
concept  was  to  have  the  government  notify  industry  of  the  required 
performance  goals  and  let  the  contractor  use  his  own  creativity  to 
design  a  weapon  system  that  will  meet  the  performance  standards. 

A  few  months  ago,  I  had  the  opportunity  to  serve  on  the  Source 
Selection  Evaluation  Board  (SSEB)  for  the  LHX  engine.  My  specific 
assignment  was  to  head  the  MANPRINT  evaluation  group  which  reported  to 
the  Integrated  Logistics  Support  (ILS)  evaluation  section.  This  was  the 
first  time  that  a  MANPRINT  specification  had  been  issued  by  the  Army  to 
industry.  Many  of  my  comments  are  geared  towards  an  analysis  of  what 
happened  when  MANPRINT  and  the  logisticians  met. 

Up  to  this  point,  our  experience  with  MANPRINT  had  been  on  a 
conceptual  level.  With  the  arrival  of  LHX,  this  concept  was  translated 
into  reality.  Like  other  translations,  some  things  translate  more 
easily  than  others  and  although  we  anticipated  conceptual  challenges,  we 
found  problems  of  acceptance  instead.  Over  the  four  months  that  the 
SSEB  convened,  the  logisticians  certainly  had  ample  opportunity  to 
challenge  the  conceptual  foundaton  of  MANPRINT.  They  didn't.  I  believe 
that  this  lack  of  sustainable  objections,  in  effect,  validates  the 
MANPRINT  concepts.  MANPRINT  was  asking  the  ILS  community  to  take  a 
different  look  at  themselves.  Call  it  resistance  to  change,  culture 
shock,  or  any  other  name,  the  potential  for  problems  was  there  and  had 
to  be  dealt  with.  The  key  to  the  problem  Is  to  understand  the  true 
nature  of  resistance.  Actually,  what  people  resist  is  usually  not 
technical  change,  but  social  change  -  the  change  in  their  human 
relationships  that  generally  accompany  technical  change.  Resistance  is 
usually  created  because  of  certain  blind  spots  and  attitudes  which 
people  have  as  a  result  of  their  preoccupation  with  the  technical 
aspects  of  new  ideas.  It  is  created  by  the  way  those  affected  by  the 
change  think  it  will  alter  their  established  relationships  in  the 
organization.  In  any  case  the  list  of  embarrassing  comments  awaits  our 
a  t  ten  tion . 


V-2 


1.  MANPRINT  IS  NOTHING  MORE  THAN  THE  APPLICATION  OF  COMMON  SENSE. 


To  those  that  believe  MANPRINT  is  nothing  more  than  the  application 
of  common  sense,  it  follows  that  almost  anyone  can  do  it.  This  thought 
showed  itself  on  both  the  organizational  and  individual  level. 
Organizationally,  one  of  the  bidders  for  the  LHX  engine  proposed  placing 
MANPRINT  as  a  subelement  of  " Service  Engineering".  Individually,  none 
of  the  personnel  chosen  by  the  three  bidders  to  lead  the  logistics 
effort  showed  any  HMPT  experience  or  training.  One  bidder,  obviously 
sensitive  to  the  increased  emphasis  on  the  MANPRINT  program,  proposed  to 
use  a  "MANPRINT  Coordinator".  Unfortunately,  the  person  nominated  to 
fill  this  position  had  a  Bachelor  of  Science  degree  in  Drama  and  no  HMPT 
background . 

There  are  basically  two  different  ways  to  deal  with  this  comment. 
First,  it  can  be  argued  that  it  is  true  -  that  a  certain  amount  of  work 
in  MANPRINT  does  reflect  the  application  of  common  sense.  Nevertheless, 
somehow  without  the  impact  of  MANPRINT  and  HMPT  specialists,  this  kind 
of  common  sense  seems  to  be  overlooked  often.  Everyone  has  his  own 

favorite  example  of  this  phenomenon.  Everyone  would  probably  agree  that 
it  would  be  illogical  for  the  military  to  develop  a  manned  weapon  system 
composed  of  hardware  and  software  for  which  someone  in  headquarters  will 
somehow  find  the  right  people  to  operate  and  maintain  it.  Common  sense 
tells  us  that  one  cannot  ignore  the  human  element  that  way  without 

effecting  system  performance.  HMPT  issues  must  be  considered  from  the 
early  design  stages.  Unfortunately,  this  very  thing  happens  far  too 
often.  The  principal  defect  in  the  process  is  that  new  developments  are 
viewed  as  systems-to-be  manned  instead  of  manned  systems. 

The  second  perspective  for  dealing  with  this  issue  argues  that 
almost  everything  is  common  sense  when  you  view  it  with  hindsight.  Let 
me  present  a  quote: 

"Outside  of  the  proven  impossible,  there  probably 
can  be  found  ...  no  field  where  so  much 
inventive  seed  has  been  sown  with  so  little 
return  as  in  the  attempts  of  man  to  fly 
successfully  through  the  air.  ...  It  may  be 
truly  said  that,  so  far  as  the  hope  of  a 
commercial  solution  of  the  problem  is 

concerned,  man  is  today  no  nearer  to 

fulfillment  than  he  was  ages  ago  when  he  first 
dreamed  of  flying  through  the  air.  ...  A 

calm  survey  of  certain  natural  phenomena  leads 
the  engineer  to  pronounce  all  confident 
prophecies  at  this  time  for  future  success  as 
wholly  unwarranted,  If  not  absurd." 

This  quotation  was  authored  by  Rear  Admiral  Melville,  then 
Engineering-Chief  of  the  U.S.  Navy  in  1901,  two  years  before  the  first 
successful  flight  at  Kitty  Hawk. 

It  is  certainly  true  that  after  the  fact  it  is  easy  to  make  almost 
any  result  appear  to  be  common  sense  and  to  put  those  predictions  on  a 
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quantitative  basis.  The  goal  of  much  of  MANPRINT  is  to  predict  what 
will  become  common  sense  after  the  fact. 

Yes,  some  HMPT  principles  are  common  sense  and  other  become  common 
sense  with  hindsight,  but  in  all  of  these  cases,  it  takes  a  professional 
to  understand  how  and  when  such  common  sensical  knowledge  applies. 

2.  THE  STUFF  YOU  DO  IS  TOO  ABSTRACT  TO  BE  USEFUL  TO  ME. 

I  frequently  heard  the  criticism  that  "MANPRINT  stuff"  is  too 
" researchy" .  I  don’t  think  one  can  expect  everything  to  be  useful 
immediately.  It  certainly  would  have  been  nice  if  MANPRINT  was  fully 
developed  and  provided  a  "recipe  book"  to  follow  that  would  insure 
adequate  emphasis  on  the  opera  tor/main  tainer  of  the  system.  Although 
many  players  in  the  MANPRINT  community  are  actively  working  towards  that 
goal,  we  are  not  there  yet.  One  purpose  of  research  is  to  advance  the 
state  of  knowledge.  This  Is  a  gradual  process.  The  individual  who 

makes  this  comment  needs  to  understand  that  it  takes  time  to  see 

MANPRINT  issues  pay  off.  Those  in  the  logistics  community  should  keep 
this  in  mind.  On  the  other  hand,  we  in  the  HMPT  community  need  to 
reali.ze  that  logisticians  have  immediate  problems  in  the  9-5  working 
world  that  need  Immediate  attention. 

Because  MANPRINT  is  not  yet  a  fully  developed  technical  program, 
Industry  was  somewhat  mystified  by  its  requirements.  This  uncertainty 
manifested  Itself  in  the  contractor’s  proposals.  It  was  Interesting  to 
note  that  some  of  the  bidders  chose  to  use  non-binding  contractual/legal 
terms  when  dealing  with  MANPRINT  issues.  Whereas  the  remainder  of  the 
proposal  used  the  contractually  binding  term  "shall",  the  MANPRINT 
section  used  the  term  "will".  The  bidders  were  wary  of  being  held 

contractually  responsible  for  something  they  did  not  fully  understand. 
We,  as  government  employees,  must  make  every  effort  not  only  to 

de-mystify  MANPRINT,  but  to  Insure  that  the  appropriately  binding  terms 
are  used.  Like  the  government,  contractors  also  would  have  appreciated 
the  security  of  a  "recipe  book"  that  would  have  provided  guidance.  In 
the  absence  of  such  a  guide,  it  is  a  good  idea  to  provide  contractors 
with  a  listing  of  the  key  points  of  contact  thus  enabling  them  to  talk 
to  the  MANPRINT  players  before  they  formulate  their  MANPRINT  plans.  This 
will  enable  contractors  to  begin  the  familiarization  process  that  can 
change  their  perception  of  MANPRINT  from  the  abstract  to  the  real  world. 

3.  THAT'S  A  GREAT  CONCEPT,  BUT  YOU  JUST  CAN'T  DO  IT  THAT  WAY. 

While  serving  on  the  SSEB,  I  had  the  opportunity  to  present  a 
number  of  briefings  on  MANPRINT's  approach  to  the  LSA/LSAR  process,  the 
concept  of  "skill"  and  the  use  of  MOSs  to  set  MPT  requirements.  The 
audiences  listened  attentively  and  asked  pertinent  questions.  The 
discussions  always  started  with  abstract  philosophy  and  gradually 
shifted  to  practical  applications.  I  was  told  how  interesting  and 
sensible  MANPRINT  was,  and  yet,  time  after  time,  I  was  told  that  it  was 
best  just  to  conduct  business  as  usual.  "There  was  no  need  to  propose 
new  ways  ‘to  ascertain  HMPT  requirements."  After  all,  the  "LSA/LSAR 
process  already  addresses  the  MANPRINT  elements",  "skill  Is  known  and 

stable  over  time",  and  "just  use  the  MOS  to  determine  MPT  limitations." 
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I  was  told  that  these  approaches/definitions  had  worked  in  the  past, 
everyone  was  familiar  with  them,  and  changing  anything  would  be  costly 
and  counterproductive,  A  lot  of  resistance  can  be  hidden  under  the 
statement  of  "that's  a  great  concept,  but  you  just  can't  do  it  that 
way.” 

This  is  indeed  a  frequent  criticism  of  the  MANPRINT  specialist 
working  in  an  engineering  environment,  but  let  the  logistician  be 
patient  with  us.  Any  specialist  worth  his  salt  will  learn  rapidly  what 
the  technical,  engineering  design,  cost  and  logistical  constraints  are. 
Our  field  depends  on  the  ability  to  make  trade-offs  between  MANPRINT 
Issues  on  the  one  hand  and  technical,  design,  cost  and  logistical 
constraints  on  the  other. 

The  logistician  will  also  want  to  be  careful  when  he  uses  this 
gambit  that  he  is  not  exposing  his  own  flank.  MANPRINT  specialists  can 
also  say,  "that's  a  fine  design,  but  the  opera  tor/ main  tainer  just  won't 
use  it  that  way". 

Finially,  it  is  ironic  that  this  phrase  was  used  by  some  to  resist 
MANPRINT.  The  RFP  procurement  process  presented  me  with  the  opportunity 
to  use  this  statement  to  show  members  of  the  Project  Managers  Office 
(PMO)  and  the  SSEB  how  a  great  concept  (for  example  -  a  stream-lined, 
Intergrated  RFP)  can  prove  detrimental  to  the  achievement  of  your  goals 
(in  this  case  -  to  shorten  the  procurement  process  and  to  produce  an  RFP 
that  will  convey  clearly  the  MANPRINT  requirements).  The  LHX  Engine  RFP 
(written  by  the  PMO)  had  three  sections  that  dealt  with  MANPRINT:  1) 
Section  L  -  Instructions,  Conditions  and  Notices  to  Offerors,  2)  Section 
M  -  Evaluation  Factors  for  Award  and  3)  Attachment  1  -  System 
Specification.  Section  L  had  the  word  MANPRINT  as  a  paragraph  title 
with  no  supporting  prose.  Section  M  stated  that  "the  MANPRINT  factor 
will  be  evaluated  based  on  the  adequacy,  completeness  and  level  of 
detail  provided  in  describing  how  the  offeror  intends  to  fulfill  the 
stated  System  Specification  requirements.  Attachment  1  not  only  did 
not  single  out  any  MANPRINT  specifications,  but  the  government's  other 
system  specifications  more  often  than  not  violated  the  very  MANPRINT 
principles  for  which  the  contractor  should  be  held  accountable.  It  is 
easy  to  understand  why  the  offerors  had  trouble  writing  coherent 
proposals  on  a  topic  which  had  no  requirements.  As  they  went  from  the 
RFP  to  the  Evaluation  Factors  to  the  System  Specifications,  they  were 
being  led  along  a  path  that  led  nowhere.  While  I  was  assigned  to  the 
board,  a  significant  amount  of  my  time  was  spent  trying  to  rectify  this 
situation.  Numerous  times  barriers  were  placed  in  the  way  of  obtaining 
an  RFP  containing  suitable  MANPRINT  requirements.  The  MANPRINT 
evaluation  group  proposed  to  remedy  the  situation  by  preparing  an  RFP 
amendment  containing  MANPRINT  requirements.  Three  versions  of  the 
amendment  was  submitted  to  SSEB  management  in  an  effort  to  produce  an 
amendment  that  was  acceptable  to  them.  Three  times  the  RFP  amendment 
was  re-drafted  to  comply  with  their  special  tailoring  guidance.  Three 
times  these  proposals  were  rejected.  We  gave  MANPRINT  briefings, 
prepared  specially  tailored  questions  for  offerors  to  elicit  suitable 
technical  responses  (which  could  be  evaluated),  and  worked  with  other 
technical  specialty  programs  (e.g.,  LSAR,  HFE)  to  incorporate  MANPRINT 
concerns.  (While  upper  management  was  sometimes  resistant  to  change, 
LSAR  &  HFE  are  two  examples  where  once  they  gained  some  familiarity  with 
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MANPRINT,  lower  level  technicians  assisted  in  integrating  MANPRINT 
concerns  into  the  weapon  systems  acquisition  process)*  When  these 
efforts  produced  less- than-op timum  responses  by  offerors,  still  another 
request  for  an  RFP  amendment  was  made.  By  carefully  marshalling  our 

arguments  and  showing  the  technical  deficits  caused  by  the  present 

course,  the  MANPRINT  evaluation  group  was  able  to  convince  three  levels 
of  SSEB  management  to  make  an  eleventh  hour  change  of  direction  and 
include  in  an  RFP  amendment  the  basic  substance  of  MANPRINT.  A 

stream-lined,  integrated  RFP  is  a  good  idea.  However,  in  this  case,  the 
integration  effort  almost  didn't  work.  It  was  only  by  persistance  that 
the  barriers  were  broken. 

4.  MANPRINT  ISN'T  THAT  IMPORTANT  -  PEOPLE  ARE  ADAPTIVE 

While  I  have  heard  this  point  argued  many  times,  I  shouldn't  have 

to  belabor  it  with  this  audience.  It's  a  well-known  fact  that  people 
using  equipment  designed  with  integrated  HMPT  elements  are  more 
resistant  to  errors.  It  is  true  that  people  are  adaptable  and  that, 
after  experience  with  a  system,  they  have  more  difficulty  reporting  on 
its  shortcomings,  but  that  doesn't  mean  that  their  efficiency  and 
effectiveness  is  not  reduced. 

The  education  of  HMPT  specialists  wouldn't  be  complete  if  they  were 
not  exposed  to  the  classic  story  of  an  ingenious  experiment  reportedly 
conducted  at  the  Bell  Telephone  Laboratory  during  World  War  II: 

The  telephone  company  was  concerned  about  the  ways  to 
save  copper  (a  scarce  commodity  during  war)  and 
one  use  of  copper  was  in  the  cable  connecting  the 
telephone  handset  to  the  table- top  set.  The 

experimental  question  was,  "How  long  does  a 
telephone  cord  need  to  be?"  The  investigator 
identified  a  series  of  test  telephones  around  the 
laboratory.  Each  night,  they  went  to  those 
telephones,  reduced  the  length  of  the  line  cord 
by  one  inch  and  replaced  the  telephone  in  its 
normal  position  on  the  desk.  They  then 

designated  a  special  telephone  operator  to 

receive  complaints  about  the  telephone.  Day  by 
day,  the  line  cords  got  shorter,  and  one  by  one, 
their  users  began  to  complain  to  the  special 
operator.  When  the  line  cord  had  reached  a  given 
length,  about  half  of  the  participants  had 

complained,  and  this  process  continued  until  the 
line  cord  was  only  a  few  inches  long.  There 

remained  one  person  who  still  had  not  complained. 

The  investigators  decided  to  check  up  on  him.  One 
of  them  visited  his  office  while  the  second 

called  him  on  the  telephone.  Sure  enough,  when 

the  telephone  rang,  he  leaned  over  his  desk  so 

that  his  ear  could  reach  the  handset,  and  he 

answered  the  phone  In  a  very  awkward  manner 

indeed.  When  asked  later  If  there  was  anything 
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funny  about  his  telephone,  he  said,  "Oh,  the  line 
cord  is  a  little  short,  but  that  doesn't  concern 

t* 

me  . 

As  I  said,  people  are  very  adaptive,  but  that  doesn't  justify  poor 
design,  training  or  performance. 

5.  AFTER  I  GET  THE  LOGISTICS  SYSTEM  WORKING,  THEN  I'LL  LOOK  INTO 
MANPRINT  ISSUES  ...  IF  I  HAVE  ANY  TIME  LEFT. 

This  assertion  is  all  too  often  offered.  We  (in  the  MANPRINT 
community)  are  plagued  with  being  left  out,  or  with  getting  in  after  the 
major  decisions  have  been  made.  By  getting  in  early,  we  can  have  orders 
of  magnitude  more  influence  on  system  performance;  but,  when  getting  in 
late,  our  influence  is  reduced  to  percentage  points.  Getting  in  early 
depends  on  management,  and  that  leads  me  back  to  the  RFP. 

It  is  not  possible  to  over-stress  the  importance  of  the  RFP.  The 
draft  RFP  for  the  LHX  engine  was  511  pages.  At  the  direction  of  the 

Under  Secretary  of  the  Army,  the  RFP  was  reduced  75%  to  a  total  of  144 
pages  to  make  it  more  manageable.  Unfortunately,  crucial  details  were 
cut  as  well.  MANPRINT's  original  input  went  from  12  sentences  to  1 
word.  MANPRINT  appeared  as  a  paragraph  title,  but  none  of  the  prose 
under  the  title  had  anything  to  do  with  MANPRINT.  No  longer  were  there 
any  MANPRINT  requirements  in  the  government's  RFP.  (Other  sections 
experienced  a  similar  fate  -  i.e.,  the  LSA/LSAR  Provisioning  input  was 
reduced  from  33  pages  to  1  paragraph,  the  HFE  section  experienced  a  20% 
reduc  tion ,  e  tc . ) . 

In  attempting  to  reduce  the  bulk  of  the  RFP  and  because  time  was 

not  allowed  for  coordination  with  the  suppliers  of  the  original 
submissions,  the  prose  reductions  were  made  by  members  of  the  PMO  who 
possessed  inadequate  knowledge  of  these  disciplines  and  could  not 
appreciate  the  full  ramifications  of  their  editorial  changes.  Theirs 
was  simply  an  honest  attempt  to  comply  with  the  guidance  they  received. 
Not  surprisingly,  the  topic  areas  that  they  were  least  familiar  with 
suffered  the  most.  Some  of  the  problems  encountered  by  the  MANPRINT 

evaluation  group  with  this  RFP  could  have  been  minimized  or  avoided  if 
the  time  for  proper  coordination  with  subject  matter  experts  had  been 

taken  at  the  beginning  of  the  process  before  multiple  changes  were  made. 

This  management  approach  of  trying  to  fine-tune  MANPRINT  issues 
later  leads  us  to  the  next  Issue. 

6.  DON'T  WORRY  -  WE  CAN  ALWAYS  MODIFY  THE  RFP  AND/OR  NEGOTIATE  A  CHANGE 
WHILE  WE'RE  EVALUATING  THE  PROPOSALS. 

It  sounds  so  logical  .  .  .  simply  modify  the  RFP  .  .  .  why  delay 

the  process?  Certainly  It  is  appropriate  to  modify  a  RFP  where  a  change 
will  benefit  the  government.  The  problem  arises  In  a  situation  where 
the  various  disciplines  have  differing  opinions  as  to  what  will  benefit 
the  government.  For  the  LHX  engine,  the  logisticians  were  only  the 
first  group  that  had  to  be  convinced  that  MANPRINT  concerns  needed  to  be 
in  the  RFP  and  were  too  Important  to  leave  to  the  vagueness  of  the 
after-contract-award  negotiation.  By  showing  SSEB  management  the 
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technical  deficits  caused  by  the  present  RFP,  management  was  convinced 
of  the  benefits  of  change.  We  have  already  discussed  the  sequence  of 
events  that  led  to  an  RFP  amendment  concerning  MANPRINT  requirements 
(see  item  //3,  page  4).  Once  this  amendment  was  accomplished,  the  legal 
and  contracting  people  needed  to  be  persuaded  as  well. 

At  this  point,  we  should  guard  against  complacency.  Keep  in  mind 
the  criticality  of  the  RFP.  Just  when  you  think  you  have  all  the 
elements  under  control,  MMurphyM  strikes.  To  understand  this  situation, 
you  must  understand  the  document  structure.  The  government 

documentation  consists  of  the  RFP,  an  Evaluation  Plan  and  the  System 
Specification.  Parts  of  industry's  response  were  proposals  for  System 
Specifications,  an  Integrated  Logistics  Support  Plan,  Human  Factors 
Plan,  MANPRINT  Plan,  etc.  Industry* s  System  Specification  was  a  general 
statement  of  what  the  contractor  would  deliver  and  was  not  generally 
distributed  to  all  of  the  government  evaluators.  The  government  *s 
legal  advisor  (Murphy  in  disguise),  notified  the  SSEB  that  the 
contractor’s  System  Specification  takes  contractual  precedence  over  the 
individual  technical  plans  for  proposal  evaluation  purposes.  The 
problem  occurs  when  industry’s  System  Specification  "parrots  back"  the 
RFP  requirements,  but  their  technical  plans  exclude  the  specific  items 
or  details  required  to  satisfactorily  complete  the  entire  System 
Specification  requirement.  Due  to  this  contractal  precedence,  the 
contractor’s  technical  plans  can  contain  numerous  weak  points,  be  of 
marginal  value,  and  yet  receive  a  satisfactory  rating.  It  is  only  when 
the  proposal  contains  an  inadequacy  or  omission  that  will  preclude  the 
contractor  from  meeting  program  objectives  that  we  were  allowed  to  rate 
a  proposal  as  unsatisfactory. 

Confusing  isn’t  it?  Here’s  an  example  of  the  government’s  legal 
advisors  helping  industry  meet  the  letter  of  the  law  rather  than  the 
intent.  Somehow  I  don't  believe  that  this  is  what  Congress  had  in  mind. 
Looking  at  it  another  way,  the  offeror  is  judged  satisfactory  if  he 
states  that  he  merely  intends  to  meet  the  requirements.  This  is 
analogous  to  hiring  someone  who  agrees  to  meet  the  requirements  of  a  job 
description  and  yet  not  being  allowed  to  evaluate  his  training, 
education,  experience,  etc. 

The  fix  for  this  predicament  is  to  make  the  offer's  System 
Specification  an  ungraded  requirement  that  must  be  met  prior  to  the 
evaluation  of  technical  plans.  That  is,  if  the  System  Specification  is 
not  responsive  to  the  requirements  of  the  RFP,  the  detailed  plans  are 
not  evaluated. 

So  much  for  the  six  potential  embarrassment  and  lessons  learned.  I 
hope  I  have  convinced  you  that  we  need  no  longer  be  defensive  about 
them.  Although  we  were  curious  as  to  whether  MANPRINT  would  attract 

any  conceptual  challenges,  none  were  offered.  Problems  of  acceptance 
and  resistance  to  change  by  some  were  the  only  obstacles  we  encountered. 
The  absence  of  conceptual  disagreements  validates  the  underpinnings  of 
the  MANPRINT  program. In  my  view,  we  are  currently  in  a  phase  where  HMPT 
work  is  receiving  increasing  acceptance.  Military  budgets  for  human 
resources  research  and  development  continue  to  rise. 

In  short,  I  have  never  been  more  enthusiastic  about  MANPRINT  issues 
and  it's  potential  for  serving  as  one  interface  between  technology  and 
society.  Some  of  the  relatively  young  logisticians  were  willing  to 
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listen.  Some  were  willing  to  take  a  chance  and  learn.  We  need  to  teach 
them.  In  time,  the  others  will  follow.  With  experience  and  lessons 
learned  as  a  basic  foundation,  we  can  achieve  realistically  stated 
system  performance  goals  during  system  development  while  reducing 
manpower  and  personnel  demands.  I  hope  you  share  my  enthusiasm. 
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U.S.  ARMY 


RELIABILITY 


PERSONNEL 


LOGISTICS 

ENGINEERING 


HUMAN 

FACTOR 

ENGINEERING 


TRAINING  SAFETY 


THEIR  VIEW... 


U.S.  ARMY  RELIABILITY 


OUR  VIEW... 
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Manprint  Coordinator 

Respormible  for  Implementing  the 
TIOO  MXNPRINT  ProgTam  Plan, 
which  includes  coordinating  Ail 
activities  and  integrating  human 
raaource  requirements  for  man¬ 
power  quantity,  per  vonnel  quaJity, 
and  skill  training  requirements  into 
the  TIOO  design 


Two  years  experience  atHHi  n  logistics  field.  Three  yean  industry  experience  in 
technical  writing,  where  he  developed  work  package  programs  for  military  applica¬ 
tions  In  addition  Mr,  (MU  has  four  yean  of  Marine  Corps  experience  in  maintenance 
and  training  He  is  currently  as  Arizona  Army  NatlonaJ  Guard  rotary  wing  pilot, 

R.S.  Acting  and  Directing  Northern  Arizona  University 
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RFP 


"AFTER  I  GET  THE 
LOGISTICS  SYSTEM 
WORKING,  I'LL  LOOK  INTO 
MANPRINT  ISSUES. .  .IF  I 
HAVE  ANY  TIME  LEFT." 
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ACQUISITION  DOCUMENTS.... 
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ATTACHMENT  W 

STATE-OF-THE-ART  IN  AI/ES,  DECISION  AIDING  AND 


AUGMENTATION  FOR  ADVANCED  MILITARY  SYSTEMS 


DOD  HFE  TAG  -  FALL  1985 


ARTIFICIAL  INTELLIGENCE  and  EXPERT  SYSTEMS 

FOR 

ADVANCED  MILITARY  and  SPACE  SYSTEMS 


THOMAS  M.  COOK 
Martin  Marietta 
Denver  Aerospace 


PURPOSE 


TO  PROVIDE  A  SUMMARY  OF  CURRENT  ”  STATE-OF-THE-ART- 
REGARDING  THE  APPLICATION  OF  Al,  EXPERTS  SYSTEMS  AND 
"SMART”  DECISION  AIDS  TO  DOD  and  NASA  DOMAINS. 

IQE1QS.  TO  BE  COVERED  INCLUDE: 

•  A  BRIEF  DESCRIPTION  OF  THE  TECHNOLOGY  AREAS 

•  TYPICAL  APPLICATION  AREAS 

•  CURRENT  RESOURCES  (e.g.t  Personnel,  HW,  SW,  etc.) 

•  CURRENT  STATE  OF  TECHNOLOGY  READINESS 

•  FUTURE  DIRECTIONS  &  CHALLENGES 
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TECHNOLOGY  AREAS 


ARTIFICIAL  INTELLIGENCE:  The  attempt  to  produce  computer  systems  (l.e., 
HW,  SW,  I/O  devices,  etc.  which  can  perform  complex  tasks  requiring 
reasoning  and  perception.  That  Is,  to  produce  machines  which  can  ”  act 
Intelligently''  ,  to  Include  the  enhancement  of  human  Intelligence. 

related  AREA?; 

EXPERT  SYSTEMS 

NATURAL  LANGUAGE  UNDERSTANDING 
IMAGE  UNDERSTANDING 
INTELLIGENT  ROBOTICS 

ADVANCED  COMPUTER  ARCHITECTURES  (E.G.,  Butterlly, 
Connection  Machine,  etc.) 

INTELLIGENT  DECISION  AIDS 


EXPERT  SYSTEMS:  Computer  programs  (and  associated  Interfaces)  which  are 
designed  for  high  performance  .  That  Is,  a  degree  of  functionality  and 
and  efficiency  comparable  with  or  better  than  human  experts. 


TYPICAL  APPLICATION  AREAS  FOR  Al  and  EXPERT  SYSTEMS 


POD  APPLICATIONS 

DARPA  STRATEGIC  COMPUTING  PROGRAM 

•  Autonomous  Land  Vehicle 

•  Battle  management  (C3I,  Staff  Planning,  etc.) 

•  Pilots  Associate 

GENERIC  APPLICATIONS  (e.g.,  Army  R&D  Plan) 

•  Human/  Equipment  Interface  Aids 

•  Planning  and  Monitoring  Aids 

•  Expert  Advisors 

•  Data  Access  and  Assimilation  Aids 

•  Support  Systems 

•  Situation  Assessment  Systems 

•  System  Controllers 

•  Information  Collectors 

•  Training  &  Instruction  (e.g.,  ICAI) 
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TYPICAL  APPLICATION  AREAS  FOR  Al  and  EXPERT  SYSTEMS  (Cont.) 


NASA  APPLICATIONS 


SPACE  STATION  PROGRAM 


•  System  Health  &  Status  Monitoring 

•  Fault  Detection,  Isolation  and  Control 

•  Failure  Prediction  (e.g.,  Trend  Analysis) 

•  Resource  Management 

•  On-Orbit  Maintenance  &  Repair  Assistance 

•  Experiment  Assistance 

•  Crew  Training 

•  Launch  Processing  &  Status 

•  Mission  Planning 

•  Operations  Control 

OTHER 

•  Satellite  Monitoring  and  Control 

•  Intelligent  Robotic  Planners 

•  Intelligent  Autonomous  Spacecraft 


RESOURCES 


Personnel 


•  Shortage  of  skilled  Knowledge  Engineers 

(Computer  scientists,  psychologists,  HFEs,  other  ?) 

•  Difficulty  isolating,  capturing  and  representing  true  MexpertisoM 

•  Salaries  and  benefits  (  or,  can  DOD  and  NASA  "Grow  their  own"  ?) 

Hardware  &  Software  Environments 

•  Al  (e.g.,  LISP)  development  machines  available  and  DECREASING 
In  size  and  price  while  INCREASING  significantly  In  speed  and 
flexibility. 

•  Al  application  machines  (e.g.,  LISP  machine  on  a  chip)  available 
within  1-3  years  (raditlon  hardened,  SEU  Immune,  etc.). 

•  Powerful  software  development  tools  available  for  all  levels  of 
application,  to  Include  Intelligent  graphics  (e.g.,  ART,  KEE,  OPS5,  etc.). 
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TECHNOLOGY  READINESS  ASSESSMENT 


#  EXPERT  SYSTEMS 

LISP  MACHINES  (QUAl.) 
HOL  FOR  DEVELOPMENT 
SW  DEVELOPMENT  TOOLS 
HIGH  RES  DISPLAY8 
-REAL  TIME-  PROCE8SINO 
S3  APPLICATIONS  S/W 


PRINCIPALS  UNDERSTOOD  -  , 

CONCEPTUAL  DESIGN  FORMULATEO  -  1 

CONCEPTUAL  OESIGN  TESTEO  - - — 

CRITICAL  FUNCTIONS  OEMON8TRATEO - 

8REAODOARO  TESTEO - 

PROTOTYPE  TESTEO  - 

ENGINEERING  MOOEL  SPACE  TESTEO  - 

OPERATIONAL  - 


REQUIRED 
KNO  CY  M 


■  Y 


REQUIRED  BY  FOR 


CURRENT  CHALLENGES  &  FUTURE  DIRECTIONS 


ARTIFICIAL  INTELLIGENCE 

•  Natural  Language  Understanding 

•  Object/  Pattern  Recognition 

•  Machine  "Learning” 

•  Measures  of  Confidence  re:  Autonomy 


EXPERT  SYSTEMS  (a  series  of  questions) 

•  Appropriate  role  for  ES 

•  Performance  Requirements 

•  Measures  of  Performance 

•  Nature  of  "Expertise" 

•  Access  to  Experts 

•  Knowledge  Representation 

•  Verification  &  Validation  Methodology 
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ATTACHMENT  X 


CREW  SYSTEM  ERGONOMICS  INFORMATION 


ANALYSIS  CENTER  (CSERIAC) 


CREW  SYSTEM  ERGONOMICS 
INFORMATION  ANALYSIS  CENTER 


DEFINITION 


v>\ 

OERIAC 


ERGONOMICS 

INTEGRATIVE  DISCIPLINE  DEVOTED  TO 
UNDERSTANDING  AND  QUANTIFYING  HUMAN 
PHYSIOLOGICAL  AND  BEHAVIORAL  INTERACTION 
WITH  EQUIPMENT  AND  SYSTEMS 


BACKGROUND 


•  MUCH  PERTINENT  ERGONOMICS  DATA  EXIST 

•  INFORMATION  IS  WIDELY  DISPERSED  AND  NOT  READILY 
ACCESSIBLE 

•  RESPONSIVE  DoD  FOCAL  POINTS  FOR  UP-TO-DATE 
ERGONOMICS  DATA  ARE  NEEDED 


PROPOSED 


SHORT  TERM: 

•  AAMRL  HOSTS  DoD  INFORMATION  ANALYSIS  CENTER 
FOCUSED  ON  CREW  SYSTEM  ERGONOMICS  FOR 
MILITARY  AIRCRAFT  AND  SPACE  VEHICLES 

LONG  TERM: 

•  CSERIAC  PROVIDES  MODEL  AND  EXPERIENCE  BASE  TO 
SUPPORT  DEVELOPMENT  OF  OTHER  SPECIALIZED 
HF  lACs  HOSTED  BY  DoD  AGENCIES 
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OE^IAC 


FUNCTIONS  AND  SERVICES 


•  RAPID  EXPERT  CONSULTATION  AND  SUPPORT 

•  AUTHOR  COMPENDIUMS/HANDBOOKS  OF  SELECTED 
ERGONOMICS  DATA 

•  PROBLEM-SOLVING  AND  ANALYSIS 

•  MIL-STANDARDS  AND  SPECIFICATIONS 

•  IDENTIFY  AND  REPORT  HIGH-VALUE  TECHNICAL  “GAPS” 

•  RECOGNIZED  LIAISON  WITH  USER  COMMUNITIES 
—  PUBLISH  NEWSLETTERS  AND  JOURNALS 

•  PLANNING  AND  CONDUCT  OF  WORKSHOPS, 
SYMPOSIUMS,  SHORT  COURSES  AND  CONFERENCES 


DLA/DTIC  ADMINISTERED 
DoD  INFORMATION  ANALYSIS  CENTERS 

INFORMATION  ANALYSIS  CENTER 

CHEMICAL  PROPULSION  INFORMATION  AGENCY 
DATA  AND  ANALYSIS  CENTER  FOR  SOFTWARE 
INFRARED  INFORMATION  AND  ANALYSIS  CENTER 
METALS  AND  CERAMICS  INFORMATION  CENTER 
NONDESTRUCTIVE  TESTING  INFORMATION  ANALYSIS  CENTER 
RELIABILITY  ANALYSIS  CENTER 

TACTICAL  WEAPONS  GUIDANCE  AND  CONTROL  INFORMATION  ANALYSIS 
CENTER 

THERMOPHYSICAL  AND  ELECTRONIC  PROPERTIES 
METAL- MATRIX  COMPOSITES  INFORMATION  ANALYSIS  CENTER 
MANUFACTURING  TECHNOLOGY  INFORMATION  ANALYSIS  CENTER 
SURVIVABILITY/VULNERABILITY  INFORMATION  ANALYSIS  CENTER 
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POTENTIAL  USERS 


GOVERNMENT 


•  LABORATORIES 

•  PM’S  &  SPO’S 

•  EVALUATION  AGENCIES 

•  SYSTEMS  COMMANDS 

•  PENTAGON 


•  LOGISTICS 

•  INTELLIGENCE 

•  ADVISORY  GROUP  FOR 


AEROSPACE  R&D 


INDUSTRY 


•  PRIMES 


•  CONSULTANTS 


•  CONTROL/DISPLAY  CONTRACTORS  •  SUB -CONTRACTORS 


•  SYSTEMS  ANALYSTS 

•  OPERATION  ANALYSTS 


•  VENDORS 

•  SOFTWARE  DEVELOPERS 


SUBJECT  DOMAINS 


A.  DESIGN  METHODOLOGY 

B.  HUMAN  PERFORMANCE  CHARACTERISTICS 

C.  INFORMATION  DISPLAY  INTERFACES 

D.  MOTOR/CONTROL  SYSTEMS  CHARACTERISTICS 

E.  SPECIALIZED  DESIGN  ISSUES 
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SUBJECT  DOMAINS 


,'~G\ 

OERIAC 


A.  DESIGN  METHODOLOGY 

1.  OPERATIONS  MODELING 

2.  EVALUATION  TECHNOLOGY 

3.  CAD  SYSTEMS 

4.  PERFORMANCE  MEASUREMENT 

5.  COST/ PERFORMANCE  MODELING 

B.  HUMAN  PERFORMANCE  CHARACTERISTICS 

1.  SENSORY  FACTORS 

2.  COGNITIVE  FACTORS 

3.  MOTOR/ CONTROL  CHARACTERISTICS 

4.  ANTHROPOMETRIC  FACTORS 


C.  INFORMATION  DISPLAY  INTERFACES 

1.  QUALITY  METRICS 

2.  SPATIAL/TEMPORAL  CONFIGURATION 

3.  INFORMATION  PORTRAYAL 

D.  MOTOR/CONTROL  SYSTEMS  CHARACTERISTICS 

1.  CONTROL  DYNAMICS 

2.  PHYSICAL  CHARACTERISTICS 

3.  CONTROL/ DISPLAY  INTERACTION 

E.  SPECIALIZED  DESIGN  ISSUES 

1.  MULTI-CREW  INTERACTION 

2.  FLIGHT  VEHICLE  CHARACTERISTICS 

3.  OPERATOR  CENTERED  SYSTEMS 

4.  LIFE  CYCLE  CREW  STATION  MODIFICATIONS 

5.  ENVIRONMENTAL  ISSUES 
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JUSTIFICATION  PREPARATION 


WHO:  MONTEREY  TECHNOLOGIES 

—  DR.  ROBERT  HENNESSY 

OBJECTIVE:  ASCERTAIN  NEED  FOR  A  CENTRAL  DoD  FOCAL 
POINT  IN  CREW  STATION  ERGONOMICS 

TASKS:  -DEFINE  USERS 

-  SURVEY  USERS/ IACS 
—  I.D.  REQUIRED  FUNCTIONS  OF  CSERIAC 
—  ESTIMATES  OF  DIRECT  COST  RESOURCES 


^0\ 

£3ERIAC 


USER’S  SURVEY 


•  OBJECTIVES: 


—  ASCERTAIN  NEED 
—  I.D.  REQUIRED  FUNCTIONS 
—  PROVIDE  COST  BASIS 

•  WHO  WAS  SURVEYED? 

—  3705  QUESTIONNAIRES 

—  27%  RETURN 

DOD:  30% 
FEDERAL:  4% 
INDUSTRY/ACAD:  66% 


SHOULD  THERE  BE  CSE  CENTER7 
PERCENT  RESPONSES  (TOTAL  *  =  829) 


•  OF  THOSE  RESPONDING: 

—  97%  USE  CREW  STATION  EGRONOMICS  DATA  IN  THEIR 
WORK 
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PERCENT  i%i  RESPONSES 


USERS  SURVEY: 

(MOST  NEEDED  PRODUCTS  &  SERVICES) 


OERIAC 


•  STATE-OF-THE-ART  REVIEWS/TECHNOLOGY 
ASSESSMENTS 

•  QUICK  RESPONSES  TO  TECHNICAL  INQUIRIES 

•  CURRENT  AWARENESS  BULLETINS 


OERIAC 


USERS  SURVEY:  FREQUENCY  OF  USE 


PROJECTED  FREQUENCY  OF  USE 


•  AVERAGE  PROJECTED  USE: 

-  6  REQUESTS  PER  YEAR 

•  TOTAL  PROJECTED  USE  BY 
RESPONDENTS: 

>  4000  REQUESTS  PER  YEAR 


I  I  L>OD 

I  I  FED  NON  DOD 
[2§  Civilian 
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O  ESI  AC 


USERS  SURVEY:  WILLINGNESS  TO  PAY 

FOR  SERVICES 


•  79%  ACCEPT  A  FEE  BASIS 
FOR  SERVICE 


USE  ON  FEE  BASIS 


I  I  UOO 

□  KO  NON  OOO 
k££l  CIVILIAN 


OERIAC 


USERS  SURVEY:  SUMMARY  OF  FINDINGS 


•  97%  USE  CREW  STATION'  ERGONOMICS  DATA 

•  88%  SUPPORT  ESTABLISHMENT  OF  CSERIAC 

•  79%  WILLING  TO  PAY  FOR  USE 

•  HIGH  PROJECTED  FREQUENCY  OF  USE 
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TIMELINES/ MILESTONES 


OcTIAC 


-  AVOCACY  BRIEFINGS  THRU  CHAIN  OF  COMMAND 

NOV  84-  JUL85 

-  BEGIN  STUDY  TO  PREPARE  JUSTIFICATION 

1  DEC  84 

-  SURVEY  COMPLETE 

MAY  - JUN85 

-  JUSTIFICATION  COMPLETE 

JUN  85 

-  FORMAL  AVOCACY  COMMITMENTS 

JUL • AUG  85 

-  JUSTIFICATION  TO  OUSDR&D 

SEP  85 

—  SOW  TO  DLA 

JAN  86 

—  CSERIAC  CONTRACT  AWARD 

JAN  87 
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ATTACHMENT  Y 


VISION  IN  SPATIAL  DISORIENTATION  AND  LOSS  OF 


AIRCRAFT  ATTITUDE  OR  CONTROL  AWARENESS 
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PURPOSE  OF  BRIEFING 


r** 

f. 


•  PROVIDE  INSIGHT  ON  HOW  WE  PROCESS  INFORMATION 

•  POINT  OUT  DESIGN  FEATURES  THAT  IMPEDE  VALID 
INFORMATION  ACQUISITION  AND  APPROPRIATE  ACTION 

•  SPREAD  WORD  TO  USERS 

•  USE  IN  MISHAP  PREVENTION  AND  ANALYSIS 

•  ENLIST  YOUR  SUPPORT  FOR  PRACTICAL  IMPROVEMENTS 
IN  PRESENT  AIRCRAFT 

•  INSURE  THESE  ASPECTS  BE  CONSIDERED  IN  Tl€ 
DESIGN  OF  FUTURE  AIRCRAFT,  E.G.,  ATF 


POINTS 

* 

I  ROLE  OF  VISION  IN  SDO 

•  DESIGN  FEATURES  IMPACTING  A/C  ATTITUDE  AWARENESS 

•  IMPORTANCE  OF  ATTITUDE  INDICATOR:  SIZE/LOCATION 

•  HUD:  NOT  AN  ATTITUDE  INDICATOR 

•  PATTERN  RECOGNITION  DISPLAYS 


SPATIAL  DISORIENTATION 


FORMERLY:  MISMATCH  BETWEEN  VISION  t  BALANCE  ORGAN 

PRESENTLY:  MISMATCH  BETWEEN  VISION  t  VISION 
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TWO  MODES  OF  VISUAL  PROCESSING 


FOCAL 
IDENTIFIES 
OBJECT  RECOGNITION 
DISCRIMINATING 
EXCLUSIVELY  VISUAL 
LIGHTING  REQUIRED 
RESOLUTION  REQUIRED 
CONSCIOUS  ATTENTION 


AMBIENT 

ORIENTS 

OBJECT  QUALITY,  QUALITY  OF  SURROUNDS 
UNCRITICAL 

BALANCE,  PROPRIOCEPTION,  HEARING 
ANY  LIGHTING 
RESOLUTION  UNIMPORTANT 
REFLEX  LEVEL 


VISUAL  PROCESSING:  TWO -MODE  SYSTEM 
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ORIENTATION 


CONSEQUENCES  OF 
AMBIENT  MODE  REACTIONS 


I  DISTRACTION  POTENTIAL 
I  VECTION  ILLUSION 
•  FALSE  HORIZON 


DISTRACTION  POTENTIAL 


o  ANY  MOTION  ATTRACTS  THE  EYE 

DIFFICULT  TO  SUPPRESS 

-  REFLEX 

-  TRAINING 

SPOT  BOGIES 
AVOID  MID-AIR 

o  HEAD  MOTIONS  ABET  DISORIENTATION 
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UISUAL  CORTEX:  RECEPTORS  RESPONSIVE  TO  LINES 

HUMAN  CANNOT  TOLERATE  A  SENSE  OF  DISORIENTATION 

AMBIENT  MODE  UNCRITICAL:  ANYTHING  WITH  QUALITY  OF  "HORI 
HORIZON 

MASS  RULE:  LARGER  HORIZON  =  MORE  COMMANDING 


l ON ESS"  = 
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IMPACT  OF  SDO  ON  USAF 


1  JAN  1971  -  1  OCT  1985 


TOTAL. CIASS_A  OP$  FACTOR  m 


A-7 

85 

97 

55X 

9 

9X 

A-10 

95 

37 

82X 

9 

11X 

F-5 

22 

13 

59X 

1 

n 

F-a 

283 

139 

99X 

26 

IPX 

F-15 

99 

25 

51X 

5 

20X 

F-16 

60 

28 

97X 

9 

32X 
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INSTRUMENT, PAiyEL  tTYPICALJt 


1  LANDING  GEAR  POSITION  INDICATOR 
LIGHTS 

2  Ds  AC  CMJTE  T-HAACLE 

3  EN:*n:  EIRE  *A»MNC  LIGHT 

4  F_A=-  PCS  'ION  INGIGATOF 

5  A”T|TcDE  MD. GATOR  FAST  ERECT. 
S*  TCn 

6  GYRC  POATF  SVS'TCM 

7  accelerometer 

8  AIR  5  p£  E  D-MA  Z*~  INDICATOR 

9  ATTITUDE  INDICATOR 

10  ALTIMETER 

11  MASTER  CAUTION  LIGHT 

12  ENGINE  TACHOMETER  INDICATORS 

13  EXHAUST  CAS  TEMPERATURE 
MDICATORS 

14  HYDRAULIC  PRESSURE  MDICATORS 

15  CANOPY  UNLOCKED  WARNING  LIGHT 

16  OIL  PRESSURE  INDICATORS 


\ 


17 
16 
19 
2  C 
21 
22 
23 

2A 

25 

26 
27 

26 

29 

30 

31 

32 


FUEL  QUANTITY  INDICATORS 
NOZZLE  POSIT**  INDICATORS 
FUEL  FLO*  INDICATORS 
CABIN  ALTIMETER 


VERTICAL  VE  LOC iTY  IN:>CATOP 

turn  an:  slir  wdicato* 


BE ARtNG-DI STANCE -HEADING 

INDICATOR 

CLOCK 

COURSE  INDICATOR 


HORIZONTAL  TRIM  INDICATOR 


SIGHT  RETICLE  BRIGHTNESS 


CONTROL  KNOB 

SIGHT  FILAMENT  SELECTOR  SWITCH 
MISSILE  VOLUME  CONTROL  KNOB 
DOWNIOCK  OVERRIDE  BUTTON 
LANDING  GEAR  LEVER 


LANDING  GEAR  WARNING  SILENCE  BLT TONi 
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/. 


l 


11 

73 

17  * 

7SI_  * 

7J_ 

, 

7% 

7« 

M 

X 

31 

Jf 

3? 

e  _ 

-  — 

V  34 

U  -r 

LANTiNC  GEAR  AAPMNC  LIGHT 
RADAR  SCOPE 

CUT  AZIMUTH  AND  AZlMUTM-UCVATlON  INO'CATORS 
THREAT  DISPLAY  PANEL 
MASTER  CAUTION  LIGHT 
RADAR  ALTIMETER 
AIRSPEED  AMD  MACH  INDICATOR 
REEERERCE  SYSTEM  SE IE CT DR  SWITCH 
AniTUDE  •  IRCCTDR  INDICATOR 
MARKER  IEACON  IIG •IT 
ALTIMETER 

VERTICAL  VELOCITY  INDICATOR 


FRONT  COCKPIT  f  -U 


HR!  -OVfR»U*  uG-^S 

INTERNE  ful.  OlASt.T>  \h:  ZlTQt 
CANOr>  VRtil'Al  UNLOCK  nakOnE 
fECC  TARA  CMEC*  S*  TCH 
run  HO  *  INDICATORS 
TACMOVnERS 

IXHAUSt  GAS  TE**C  AATUAf  INDICATORS 
ARRESTING  HDD*  CONTROL  RANDLE 
WISTRUVENT  LKiMTS  INTENSITY 
ClP-Cl  :T  CHAMPS 
RiG"T  $w  t  ‘-.H 

*>  2*  v:?i  UC-T  PANE  l 

EIGHT  DA>  ClOC* 


Y-ll 


I 


I 

I 


4  MUO  Conti  at  P»n* 
i  Standby  Aji«ud»  fcr«Sc*tpr 

•  f  Fkm  fcndfcc*4o r 

7  DUAL  PC  FAJL  Wirr^  l(#«  *ed| 

•  HYD  01  WSS  W**rw*  |M) 

9  CANO rt  f*«* 

•0  R**o  Tr#^u#ncy  lr**eato* 

11  VtMiCil  Vt*ocny  kykitff 

12  CM  PTWHjff  ln*cn«i 

13  A  PM  In* c**r 

14  NooN  Smv  \nfrcmor 
16  FTIT  kn*c«c» 

16  Fv*<  Ow»n<rry  Indicator 

1 7  AA*m«atr 

18  MR*  8CN  L** 

It  An*ud*  Owocior  ln*O0ior 
30  Hofifonut  S*u#t*on  ln*c*or 
21  Radar  EO  CHdUt 
n  Auddaf  P60AI  AOJ  Knob 
23  AOA  Indic/toi 


24  Inat'umor*  Moda  S^t€l  Pan* 

26  Aypood  Kk»  >o*o  if 

26  St  o'  tl  Contr*  Pan* 

27  AotopAot  POLL  Sw*icb 
20  AUTOm.OT 

29  AtXop4ot  PITCH  S***ct> 

JO  MASTER  AAM  $««ch 

31  AIT  mi  Wnon 

32  SMS  PWR  SmkA 

33  FF  Of«T  iunon 

34  E9G  FPE  Wrmn|L^  Aa<f) 

35  TO  /LAND  CONFIG  Wt'rwa^  lift* 

36  THREAT  WARNING  CorHro*  and  »n*ca'orf 

37  94ASTW  CAUTION  L«fpM  lAmt**) 

38  THREAT  WARNING  Mxraior 

39  8cx>tt»9**f 

40  AOA  Indviar 

•«  f^rjovno  i^n 
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F-16 


RECLINED  SEAT 
SIDE  STICK  CONTROLLER 
EXCESS  POWER 


high  head  position 


UNOBSTRUCTED  CANOPY 


LOW  CANOPY  RAIL 


DROOP  NOSE 


SPECTACULAR  PERFORMANCE 


COCKPIT  FORWARD  OF  MING 


i 

EASE  OF  FLYING 


PILOT  CAN'T  SEE  ANY  OF  A/C 


LACK  OF  CUES 


4V 


FEELING  OF  CONTENTMENT 


4 


FEELING  OF  DCTAC«NT 


^ MAGIC  CARPET  EFFECT 


COMPLACENCY 


F-16  EUPHORIA 

INPLICS:  TINE  SENSE,  CROSS-CHECK.  S.A. 


LOSS  OF  S.A.  POTENTIAL:  F-16 

I  HEAD  HIGH  IN  FISHBOWL 

-  LACK  NATURAL  ATTITUDE  REFERENCES 

-  AMBIENT  MODE  BOMBARDMENT 

•  CANOPY  GLARE  &  REFLECTIONS 

-  INSTRUMENTS  -  WHITE  BAND 

-  RADAR  -  GREEN  BAND 

•  INSTRUMENT  LIGHTING 

-  DME  ON  HSI 

-  WI 


LOSS  OF  S.A.  POTENTIAL:  F-16 

•  FLIGHT  CHARACTERISTICS:  DECEPTIVE 

-  LACK  OF  AUDITORY/TACTILE  CUES 
TO  AIRSPEED 

-  DOESN'T  "TALK"  TO  THE  PILOT 

•  AERIAL  REFUELING  LIGHT  -  FALSE 
HORIZON 

•  FORMATION  LIGHTS 
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LOSS  OF  SA  POTENTIAL i  F-16 


•  HAVE  TO  FLY  WITH  EYEBALLS  - 

HAVE  TO  FLY  IT  CONSTANTLY 

•  NO  CLUE  THAT  ANYTHING  HAS  CHANGED 

•  HARD  TO  TRANSITION  OUTSIDE  TO  IN 


WHAT  DOES  THE  PILOT  NCCD  TO  RCCOCMB,  CMAT  AND  COPE  WITH  SDO, 
FALSE  HORIZONS,  NO  HORIZON? 

A  BIG  ATTITUDE  INDICATOR 

-  THE  BIGGER  THE  BETTER I 

-  IDEALLY  BIG  ENOUGH  TO  SEE  OUT  CORNER  OF 
EYE  -  AS  IN  FLYING  F*. 


IMPORTANCE  OF  THE  ATTITUDE  INDICATOR 

•  THE  HUB  OF  THE  CROSS-CHECK;  IN  IMC  70-90X  EYE-TIME 
I  PROMINENT,  LARGE,  CENTRALLY  LOCATED 
9  IDEALLY  BIG  ENOUGH  TO  SEE  OUT  CORNER  OF  EYE 

-  FACILITATE  TRANSITION  OUTSIDE  TO  INSIDE 

-  SNEAK- A- PEEK  FLYING  FORMATION 

-  TO  MAINTAIN  ATTITUDE  AWARENESS 

-  TO  SPEED  RECOGNITION  OF  UNEXPECTED  ATTITUDE 
AND  FACILITATE  COPING 
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.  AMI  (  Airspeed  Mach  Indicator) 

.  ADi  (Attitude  Director  Indicator) 

.  Stabilizer  Lock  Light  004$]  C/W 
.  Engine  Overheat  and  Fire  Warning  Lights 
.  Fire  Extinguisher  Button  1060  C/W 
.  HSI  (Horizontal  Situation  Indicator) 

.  AWI  (Altitude-Vertical  Velocity  Indicator) 
.  IFF  Caution  Light 
.  lail-Out  Light  (  {Q  only) 


All  cavtloff,  warning  and  Indicator  lights 
frt  pr»***t+d  in  dttsil  in  Figwra  t  -40 

All  armamtnt  controls  art  proaontod  In 
T.O.  1  F-1 05B-34-1  -1 . 

T.O.  1F-105G-1  Hr 
23  cockpit  configuration. 


10.  Cround  Speed  and  Drift  Angle  Indicator 

11.  Tachometer 

12.  Pressure  Ratio  Cage 


Y-16 


un 

WINDSHIIl© 
Ft  AMI 


tKJKT 
WINOSHMI 
Ft  AMI 


CtNTtl 

MNR 


CIWTER 

CONSOLE 


un 


uticai 


>4  sou 


Y-17 


STANDBY  ATTITUDE  INDICATOR  (SAD 


•  POSITION  RELATIVE  TO  ADI 

-  MISMATCH  RECOGNITION 

-  COMPOSITE  CROSS  CHECK 

•  RELIABILITY  ISSUES 


MOBLOI  WITH  HUD'S 


DIGITAL 


CLUTTERED 


-  CONFUSING 


Instrument  Modes 

ILS/NAVEEE 


■evunOft 


»T 

MO«  SAiCT 
FAA 


Y-18 


A  LW  EAT  HER  PLITEQAGE 


Mai  Til 


I 

I 

I 

« 

I 


At  the  *ocent  tbe  aircraft  entered  IMC,  tba  ADI  aould  have 
looked  socethlng  Ilka  this. 


At  the  aoffieot  of  impact,  tbe  ADI  eould  have  looked  Ilka 
tfcis.  .  Tbe  barrel  of  tba  ADI  la  less  tbao  2”  lo  dlaaetar, 
as  lo  this  flfure.  It  Is  located  between  tba  knees,  24M 
to  34"  fro*  tba  Deslcn  Eye  Point.  Could  It  be  possible 
to  confuse  these  indications,  such  as  at  olfbt,  with 
canopy  reflections? 


Note  Horlzoa  llae  subtends  an  ancle  of  only  3.4°  to 
4,8°  at  tba  aye  -  aot  particularly  coascaodlof. 
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SPATIAL  HIS  ORIENTATION 


.  INSIDIOUS  -  PILOT  UNAWARE  OF  OOD  ATTITUDE 


COMFORTABLE  -  LACK  OF  VERTIGO  OELAYS  X-CHCCJC 


.  EXPECTANCY  -  UNALERTED,  WHEN  HE  FINALLY  X-CHECKS, 
HE  NAY  SEE  WHAT  HE  EXPECTS  TO  SEE 


.  INSTRUMENT  MISREAD/MISINTERPRETED,  HE  MAY  DO  JUST  THAT 
E.G. i  FTR  HUD  ATTITUDE  DISPLAY 
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Display  Flold  of  .Regard  Comparison 


6ELL 

Hinr 

FIELD  OF  VIEW 

90* 

16’ 

MOTION 

STABLE 

DYNAMIC 

ROLL 

SENSING  BACKWARDS 

SENSING  BACKWARDS 

PITCH 

AIRCRAFT  ATTITUDE 

VELOCITY  VECTOR 

UNUSUAL  ATTITUDE 

RECOGNITION  t  COPING 

EASIER 

VERY  DIFFICULT  - 

SUBJECT  TO  MISINTERPRETATION 

DECEPTIVE  ASPECTS 

TOP  ONE-HALF  MAY  APPEAR 

UNIFORM  UNDER  CERTAIN  GiR 

TENDENCY  TO  STARE  «  MESMERIZING 

WITHOUT  PROCESSING  INFORMATION 

HUD  ATTITUDE  INDICATION  IMPROVEMENTS 


•  FPM  -  SKY  POINTER 

•  PITCH  SCALES  -  RADICALLY  CHANGE 
PATTERN  BETWEEN  (♦)  AND  (-),  AND 
WITHIN  (-) 


4- 

-d£ 


-f 


vTjrrrx  »o:irmc  Sr**  ok  rusir:  fate  rawir 
TD  6CT  UR«ICMT, 

*OiNT  FPM  TAIL 
AT  TUI  tTAZ 


y-?A 


HUD  PITCH  SCALE  SHOWING 
RADICAL  CHANGES  FROM  POSITIVE 
TO  NEGATIVE  &  WITHIN  NEGATIVE 
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HABIT  PATTERN 


•  LEARNING  IS  HARD,  BUT 
FORGETTING  IS  HARDER  YET 

t  APPLIES  TO  IOCATION  OF 

-  SWITCHES 

-  EJT  HANDLES 

-  INSTRUMENTS 


PI  LOT -VEHICLE  INTERFACE 


•  WHAT'S  HE  NEED? 

-  ATTITUDE  CUES 

-  AIRSPEED  • 

-  ALTITUDE  ' 


'v 

► 

J 


W/OUT  HAVING  TO  DWELL 
ON  INSTRUMENTS 


•  HAVE  WE  CONSIDERED  ADEQUATELY  HOW  THE  BRAIN 
&  PERCEPTUAL  SYSTEM  WORK  IN  FORMATTING  THE  PVI? 

-  PROPER  MIX  OF  PATTERN,  PICTORIAL,  COLOR. 
ORIENTATION,  ANALOG,  DIGITAL  FOR  FOCAL  t 
AMBIENT  VISUAL  MODES 


•  DOES  THE  AIRCRAFT  TALK  TO  THE  PILOT? 
-  AUDITORY,  KINESTHETIC  CUES 
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SOME  CHARACTERISTICS  OF  MAN  AS  A  SYSTEM  COMPONENT 


•  PERCEPTUAL  ABILITIES 

-  SENSITIVE  SOUND  i  LIGHT  DETECTORS 

-  GOOD  PATTERN  a  OBJECT  RECOGNITION 

-  SUBJECT  TO  HEIGHT/DISTANCE  MISJUDGMENTS, 
TUNNEL  VISION,  DISTRACTION,  ILLUSIONS 
POOR  TIME  SENSE 

•  LIMITED  PROCESSING  RATE 

•  INPUT  RATE 

•  THINKING  RATE 

•  MOTOR-OUTPUT  RATE 

LARGELY  SINGLE-CHANNEL  OPERATION 


•  ERROR  PRONE 

-  LIMITED  PRECISION 

-  SEQUENCE  STEP  OMISSIONS 

-  HABIT  PATTERN  REVERSIONS 

-  SUBJECT  TO  FATIGUE  AND  IRREGULAR  PERFORMANCE 

•  FLEXIBLE  CONTROL 

-  SELF-REPROGRAMMABLE 

-  ADAPTABLE 

-  POOR  MONITOR 

-  NEEDS  MOTIVATION 

•  REQUIRES  SPECIALIZED  LIFE  SUPPORT  EQUIPMENT 


CHARACTERISTICS  OF  MAN  RE  DISPLAYS 


I  MAN  -  PATTERN  RECOGNIZER 

MORE  INFO  ORGANIZED  VISUALLY  -  FASTER  ACQ/ 
UNDERSTANDING 

PICTURE'S  WORTH  A  1000  WORDS 

•  USUALLY  WANTS  ONLY  TO  KNOW  WHETHER  PARAMETER  HAS  CHANGED 

DIGITALS /CC - NUMFR I CS /S YMRO LS  — ►  FOCAL  MODE:  READ, 

DECODE,  INTEGRATE 

-  NO  INHERENT  TREND/LIMITATIONS  INFO 

ANALOG-^AMBIENT  a  FOCAL  MODES:  FASTER,  BOTH  TREND 
a  LIMITATIONS  INFO 

-  CAN  Bt  MISREAD 

•  ATTENTION  TRAPS:  ANY  DISPLAY  TRAPPING  PILOT'S  ATTENTION 

CAN  KILL  HIM 
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ALTITUDE  AWARENESS 


•  AUDITORY 

•  Datum-plane  tone  -  selectable 

•  predictive  gpws 

•  CALLOUT  OF  SELECTED  PARAMETERS  ON  REQUEST 

•  VISUAL 

•  ALTIMETER  -  PATTERNED  FOR  INSTANT  UNLOUI VOCAL 
APPRECIATION 

•  RADAR  ALTIMETER  -  PATTERNED  LIKEWISE 


CRITICAL  CONTROL  PARAMETERS 

•  ATTITUDE,  AIRSPEED,  ALTITUDE  ±  HEADING,  VERTICAL 
VELOCITY,  ANGLE  OF  ATTACK 

•  PRESENT  DISPLAYS  - 

REQUIRE  FOCAL  MODE  PROCESSING  TO  READ  &  DECODE 
"  MFNTAL  EFFORT  TO  INTEGRATE 

*  TIME  INVESTMENT  TO  OBTAIN  TREND  INFO 

•  MORE  MENTAL  EFFORT  FOR  LIMITATIONS/MEANING 
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PROPOSED  AIRSPEED  DISPLAY 
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pbuwb  nncTnare  ursrarr 


Y-29 


fKKID  nuns  DISPLAY 


N 


s 


TOLL  CCMPAiS 
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ATTITUDE  INDICATION  FORMATS 


F-1U  FORWARD  COCRRfT 
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F/A-18  HORNET  COCKPIT 


PERABLE  VISUAL 
NDITIONS 


FLYING  CONDITIONS  -  CENTRAL  EUROPE 
WINTER  -  TYPICAL  24  HR  PERIOD 
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NIGHT  POLE  NEEDS 


PILOT  CONSIDERATIONS 

•  FATIGUE 

•  REACTIONS  SLOWED 

•  PERCEPTIONS  IMPAIRED 

I  SUBJECT  TO  ILLUSIONS,  DISORIENTATION, 
DISTRACTION,  t  CHANNELIZED  ATTENTION 

AIRCRAFT  CONSIDERATIONS 

•  BETTER  ATTITUDE  REFERENCES  TO  INCLUDE 
LARGF  DEDICATED  ATTITUDE  DISPLAY  HIGH 
IN  CENTER  OF  INST  PANEL 

•  CRITICAL  CONTROL  PARAMETERS  FORMATTED  FOR 
INSTANT  UNEQUIVOCAL  RECOGNITION 

•  BETTER  COCKPIT/INST  LIGHTING,  LESS  CPY 
GLARE  S.  REFLECTIONS 

•  BETTER  FORMATION  UGWTING,  NO  FALSE  HORIZONS 


SUGGESTED  IMPROVEMENTS:  PRESENT  A/C 


I  HUD: 

INST  MODE  - 

*  VERTEX  POINTER  ON  FPM 

*  RADICALLY  CHANGE  PITCH  SCALE  PATTERN  FROM  ♦  TO  - 
AND  WITHIN  NEGATIVE 

*  CONSIDER  COLOR  CHANGE  AS  REDUNDANT  CUE 

*  LENGTHEN  ALL  PITCH  SCALE  LINES 

*  PATTERNED  ANALOG  FORMAT  FOR  PARAMETERS  SUCH  AS 
ASI ,  ALT,  VVI,  AOA  &  G 

•  DEDICATED  ATTITUDE  INDICATOR  HIGH  IN  CENTER  OF  INST  PANEL 

*  CONSIDER  HUD  CONTROL  PANFL  AREA  FOR  PROJECTED  IMAGE 
OR  FLAT  PANEL  DISPLAY 

*  CONSIDER  PVD 

•  IMPROVE  NON-VISUAL  CUES  TO  AIRSPEED 

•  IMPROVE  SPOT  LIGHTING 

*  CONSIDER  GOOSENECK  LAMPS 

•  REDUCE  GLARE  &  REFLECTIONS 

*  CONSIDER  GLARE  SHIELD  EXTENSIONS/BATWING  SHADES 

•  GPWS 

•  RADAR  ALTIMETER  '  '• 
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SUGGESTED  IMPROVEMENTS  -  FUTURE  A/C 


•  HUD 


-  IMPROVEMENTS  FOR  ATTITUDE  AS  BEFORE 

-  DEDICATED  ATTITUDE  INDICATOR  ON  HUD 
CONTROL  PANEL 

I  DEDICATED  AD  I /S A I  BELOW  HUD  C/P  FROM  DIFFERENT 
INFO  SOURCE 

•  OPTIMIZE  MIXTURE  OF  SENSORY  INPUTS:  INTEGRATED 
ANALOG/PATTERN  FORMATS:  AMBIENT  MODE,  HEARING  AND 
PROPRIOCEPTION 

•  CONSIDER' Ml  SSI  ON  DIVERSITY;  E.G.,  BASIC  INST  FLYING, 
NIGHT  WX  FM,  ENHANCED  NIGHT  ROLE 


AIRCRAFT  ATTITUDE  WORKSHOP  RECOMMENDATIONS 

•  WARNINGS  t  ALERTS  TO  ALTITUDE 

*  DATUM  PLANE  TONE-SELECTABLE 

*  GPWS 

*  RA 

•  ATTITUDE  DISPLAY 

*  UNMISTAKABLE,  JUST  BELOW  HUD,  BIG,  DEDICATED 

*  SAI  -  RELIABLE,  BIG,  HEAD  DOWN 

•  HUD  IMPROVEMENTS 

*  FPM  -  VERTEX  POINTER 

*  PITCH  SCALES  -  PATTERN,.  COLOR  DIFFERENCES  (♦)  TO  (-) 

•  NON- FOCAL  MODE  SENSORY  CUES 

*  AMBIENT  MODE  DISPLAYS  -  PATTERN/ANALOG  FMAT  -  ASI,  ALT 

*  AUDITORY/TACTILE  CUES  -  AIRSPEED 

*  AUDITORY  CALLOUT  OF  CERTAIN  INFO  -  ASI,  ALT,  FUEL  STATE 

•  NOISE-CANCELLING  HEADSETS 
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AIRCRAFT  ATTITUDE  WORKSHOP  RECOMMENDATIONS  'CONT'D) 


•  NIGHT  ROLE 

*  GLARE  &  REFLECTIONS 
‘  COCKPIT  LIGHTING 

*  FM  LIGHTING 

*  FALSE  HORIZONS 

•  G-LIM1TER  BY-PASS 

•  AUTO- RECOVERY 

*  PANIC  BUTTON 

*  CRASH  RESISTANT  FLIGHT  CONTROL  SYSTEM 

•  TRAINING 

*  ATTENTION 

*  VISION 

*  HUD  USE 

*  BASIC  INSTRUMENTS 

*  G-TOLERANCE 


Ml 


TO 

1 

MAINTAIN/REGAIN  ATTITUDE 
AWARENESS 

ALERT  TO  ALTITUDE 

PREVENT  CWG 


NEED 

i 

ATTITUDE  DISPLAY 

PROMINENT,  BIG,  DEDICATED 

AUDIO  TONES/GPWS 

AUTO -RECOVERY  SYSTEM 
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ATTACHMENT  Z 


MEETING  15 


ATTENDEE  LIST 


DEPARTMENT  OF  DEFENSE 
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